You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/15 00:31:10 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8318 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8318/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3236, name=commitScheduler-1695-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3196, name=commitScheduler-1667-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3234, name=commitScheduler-1692-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)




Build Log:
[...truncated 10288 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 775944 T3166 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsNew
   [junit4]   2> 775953 T3166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 775956 T3166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54976
   [junit4]   2> 775956 T3166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 775957 T3166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 775972 T3166 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 776004 T3166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 776005 T3166 oasc.CoreContainer.<init> New CoreContainer 312505862
   [junit4]   2> 776005 T3166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 776006 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 776007 T3166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 776008 T3166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 776008 T3166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 776009 T3166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 776009 T3166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 776011 T3166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 776012 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00001_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/
   [junit4]   2> 776012 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00000_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/
   [junit4]   2> 776013 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00004_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/
   [junit4]   2> 776013 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00002_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/
   [junit4]   2> 776014 T3166 oasc.CorePropertiesLocator.discoverUnder Found core 00003_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/
   [junit4]   2> 776014 T3166 oasc.CorePropertiesLocator.discover Found 5 core definitions
   [junit4]   2> 776015 T3166 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 776015 T3166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 776020 T3166 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 776021 T3166 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 776023 T3179 oasc.OneIndexer.run Starting indexing thread: 3179
   [junit4]   2> 776024 T3181 oasc.OneIndexer.run Starting indexing thread: 3181
   [junit4]   2> 776024 T3177 oasc.OneIndexer.run Starting indexing thread: 3177
   [junit4]   2> 776023 T3180 oasc.OneIndexer.run Starting indexing thread: 3180
   [junit4]   2> 776023 T3178 oasc.OneIndexer.run Starting indexing thread: 3178
   [junit4]   2> 776028 T3182 oasc.OneQuery.run Starting query thread: 3182
   [junit4]   2> 776028 T3174 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 776028 T3173 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 776028 T3172 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 776028 T3186 oasc.OneQuery.run Starting query thread: 3186
   [junit4]   2> 776028 T3185 oasc.OneQuery.run Starting query thread: 3185
   [junit4]   2> 776028 T3183 oasc.OneQuery.run Starting query thread: 3183
   [junit4]   2> 776028 T3184 oasc.OneQuery.run Starting query thread: 3184
   [junit4]   2> 776029 T3172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 776029 T3173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 776029 T3174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 776068 T3172 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776071 T3174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776074 T3173 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776096 T3172 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776096 T3172 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776099 T3174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776099 T3174 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776100 T3172 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 776103 T3174 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 776126 T3173 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776126 T3173 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776132 T3189 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 776133 T3189 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 776133 T3173 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 776135 T3172 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776137 T3172 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776137 T3172 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 776138 T3172 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 776138 T3174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776138 T3172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 776139 T3172 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 776139 T3172 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776139 T3174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776140 T3172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 776140 T3174 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 776140 T3174 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 776140 T3172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5aa4e74a
   [junit4]   2> 776141 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 776141 T3174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 776141 T3172 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6b98e68a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@722ebd57),segFN=segments_1,generation=1}
   [junit4]   2> 776142 T3174 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776142 T3172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776142 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 776143 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@55b2afd6
   [junit4]   2> 776143 T3172 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776143 T3174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@685c64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55cabb67),segFN=segments_1,generation=1}
   [junit4]   2> 776144 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776144 T3172 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776144 T3172 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776145 T3174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776145 T3172 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776145 T3174 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776146 T3174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776146 T3174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776148 T3174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776148 T3172 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776151 T3174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776151 T3173 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776151 T3174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776151 T3172 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776152 T3172 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776152 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776153 T3173 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776153 T3174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@685c64e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55cabb67),segFN=segments_1,generation=1}
   [junit4]   2> 776154 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776153 T3172 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776154 T3174 oass.SolrIndexSearcher.<init> Opening Searcher@13665fd2 main
   [junit4]   2> 776154 T3172 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6b98e68a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@722ebd57),segFN=segments_1,generation=1}
   [junit4]   2> 776155 T3172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776155 T3174 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 776153 T3173 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 776156 T3173 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 776155 T3174 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 776155 T3172 oass.SolrIndexSearcher.<init> Opening Searcher@432a3217 main
   [junit4]   2> 776155 T3194 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@13665fd2 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776156 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 776157 T3195 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@432a3217 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776157 T3173 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 776157 T3172 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 776158 T3173 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776159 T3172 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 776159 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 776160 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@32d9aa1e
   [junit4]   2> 776161 T3173 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6e44cea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3daac8ba),segFN=segments_1,generation=1}
   [junit4]   2> 776161 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776163 T3173 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> ASYNC  NEW_CORE C624 name=00003_core org.apache.solr.core.SolrCore@5e531ef7
   [junit4]   2> 776164 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 2
   [junit4]   2> 776164 T3173 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776165 T3173 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776165 T3173 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> ASYNC  NEW_CORE C625 name=00004_core org.apache.solr.core.SolrCore@50557430
   [junit4]   2> 776167 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 2
   [junit4]   2> 776167 T3173 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776169 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 1
   [junit4]   2> 776169 T3188 C624 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776170 T3173 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776170 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 776170 T3173 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776170 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 1
   [junit4]   2> 776171 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 0
   [junit4]   2> 776171 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776172 T3173 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6e44cea lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3daac8ba),segFN=segments_1,generation=1}
   [junit4]   2> 776173 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776172 T3188 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 776173 T3188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 776173 T3173 oass.SolrIndexSearcher.<init> Opening Searcher@11abeae main
   [junit4]   2> 776174 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 776174 T3173 oasc.SolrCores.putTransientCore Opening transient core 00000_core
   [junit4]   2> 776174 T3198 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@11abeae main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776174 T3173 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 776176 T3189 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776181 T3175 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C626 name=00000_core org.apache.solr.core.SolrCore@781fcd95
   [junit4]   2> 776181 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 3
   [junit4]   2> 776181 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 3
   [junit4]   2> 776184 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 776187 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> 776190 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 776192 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 776195 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 0
   [junit4]   2> 776198 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 776201 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 0
   [junit4]   2> 776207 T3189 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 776208 T3188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 776208 T3189 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776211 T3189 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 776223 T3189 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776224 T3189 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776224 T3189 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 776224 T3189 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 776225 T3189 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 776225 T3189 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 776225 T3189 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776225 T3189 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 776226 T3189 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3182e3df
   [junit4]   2> 776227 T3189 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10a84ae4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@468aca88),segFN=segments_1,generation=1}
   [junit4]   2> 776227 T3189 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776228 T3189 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776228 T3189 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776230 T3189 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776231 T3189 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776232 T3189 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776232 T3189 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776232 T3189 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10a84ae4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@468aca88),segFN=segments_1,generation=1}
   [junit4]   2> 776233 T3189 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776233 T3189 oass.SolrIndexSearcher.<init> Opening Searcher@20289fbc main
   [junit4]   2> 776234 T3189 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 776234 T3200 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@20289fbc main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776234 T3189 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 776236 T3188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> ASYNC  NEW_CORE C627 name=00001_core org.apache.solr.core.SolrCore@76ecef0a
   [junit4]   2> 776238 T3189 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776238 T3188 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 776240 T3190 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776240 T3191 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776240 T3192 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776242 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 3
   [junit4]   2> 776242 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 2
   [junit4]   2> 776243 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> 776246 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 776246 T3188 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 776247 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 1
   [junit4]   2> 776248 T3191 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 0
   [junit4]   2> 776250 T3192 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 776251 T3174 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 776252 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> 776254 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 776255 T3187 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 776256 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 0
   [junit4]   2> 776260 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 776261 T3188 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 776262 T3188 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 776263 T3188 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 776263 T3188 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 776263 T3188 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 776264 T3188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 776264 T3188 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 776264 T3188 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 776264 T3188 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2d681d4d
   [junit4]   2> 776265 T3188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@969c672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e98a567),segFN=segments_1,generation=1}
   [junit4]   2> 776265 T3188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776266 T3188 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 776266 T3188 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 776267 T3188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 776267 T3188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 776268 T3188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 776269 T3188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 776270 T3188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 776270 T3188 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595179258, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 776271 T3188 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@969c672 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e98a567),segFN=segments_1,generation=1}
   [junit4]   2> 776271 T3188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 776271 T3188 oass.SolrIndexSearcher.<init> Opening Searcher@6428b28b main
   [junit4]   2> 776271 T3172 C625 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776272 T3188 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 776272 T3202 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@6428b28b main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 776272 T3188 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> ASYNC  NEW_CORE C628 name=00002_core org.apache.solr.core.SolrCore@3ec4fbaf
   [junit4]   2> 776277 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 2
   [junit4]   2> 776281 T3175 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 3
   [junit4]   2> 776282 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 4
   [junit4]   2> 776282 T3174 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 4
   [junit4]   2> 776282 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 4
   [junit4]   2> 776283 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 2
   [junit4]   2> 776285 T3190 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 776285 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 776286 T3192 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 776286 T3172 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 776287 T3188 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 776288 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 776288 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 1
   [junit4]   2> 776289 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 776290 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> 776290 T3191 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 776291 T3190 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 1
   [junit4]   2> 776291 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 776292 T3192 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 776292 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 0
   [junit4]   2> 776293 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 776293 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 776294 T3172 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 776294 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 776295 T3173 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 776295 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 0
   [junit4]   2> 776296 T3190 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 776296 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 0
   [junit4]   2> 776297 T3192 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 776297 T3189 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 776298 T3188 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 776298 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 776299 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 776300 T3174 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 1
   [junit4]   2> 776300 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 776301 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 776301 T3190 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 776302 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 776303 T3192 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 776303 T3189 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 776304 T3188 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 776304 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 776304 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 0
   [junit4]   2> 776306 T3174 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 776306 T3173 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 0
   [junit4]   2> 776307 T3191 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 776307 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 776307 T3187 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 0
   [junit4]   2> 776309 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 776309 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 776309 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 0
   [junit4]   2> 776310 T3175 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 776310 T3192 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 776312 T3174 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 776312 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 776313 T3191 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 776313 T3190 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 776313 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 0
   [junit4]   2> 776315 T3172 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 776315 T3189 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 776315 T3188 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 0
   [junit4]   2> 776316 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 776316 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 0
   [junit4]   2> 776318 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 1
   [junit4]   2> 776318 T3174 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 776318 T3191 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 776320 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 776320 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 776321 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 776321 T3188 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 0
   [junit4]   2> 776321 T3192 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 0
   [junit4]   2> 776323 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 776324 T3189 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 2
   [junit4]   2> 776326 T3174 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 2
   [junit4]   2> 776326 T3191 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 776327 T3173 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 4
   [junit4]   2> 776327 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 776327 T3190 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 776330 T3172 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 2
   [junit4]   2> 776330 T3188 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 776331 T3175 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 776331 T3189 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 1
   [junit4]   2> 776331 T3192 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 2
   [junit4]   2> 776333 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 1
   [junit4]   2> 776335 T3191 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 2
   [junit4]   2> 776335 T3173 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> 776336 T3187 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 2
   [junit4]   2> 776336 T3190 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 1
   [junit4]   2> 776337 T3172 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 776339 T3188 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 2
   [junit4]   2> 776339 T3175 C625 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 776340 T3192 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 776339 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 776340 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 776341 T3191 C625 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776343 T3173 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 2
   [junit4]   2> 776343 T3188 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776344 T3192 C627 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 776343 T3190 C626 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 776344 T3172 C628 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> 776345 T3189 C624 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 776343 T3187 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 1
   [junit4]   2> 776346 T3174 C627 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id130]} 0 1
   [junit4]   2> 776347 T3175 C626 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 4
   [junit4]   2> 776349 T3191 C627 oas

[...truncated too long message...]

 [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=3227, name=commitScheduler-1683-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=3201, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=3232, name=commitScheduler-1688-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=3203, name=commitScheduler-1677-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=3199, name=commitScheduler-1671-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=3197, name=commitScheduler-1668-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=3233, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2801A6C607886F27]:0)
   [junit4] Completed on J0 in 54.11s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 585 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 823 ignored (5 assumptions)

Total time: 39 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8319 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8319/
Java: 32bit/jdk1.7.0_45 -client -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3366, name=commitScheduler-2444-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=3362, name=commitScheduler-2440-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=3336, name=commitScheduler-2420-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)




Build Log:
[...truncated 10376 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 978831 T3301 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 978839 T3301 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 978862 T3301 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56396
   [junit4]   2> 978863 T3301 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 978863 T3301 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 978864 T3301 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 978864 T3301 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 978883 T3301 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 978943 T3301 oasc.CoreContainer.<init> New CoreContainer 11731529
   [junit4]   2> 978943 T3301 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 978945 T3301 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 978946 T3301 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 978947 T3301 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 978947 T3301 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 978948 T3301 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 978949 T3301 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 978949 T3301 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 978983 T3311 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 978983 T3311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 979016 T3311 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979060 T3311 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979061 T3311 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979065 T3311 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 979079 T3311 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979080 T3311 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979080 T3311 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 979081 T3311 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 979081 T3311 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 979081 T3311 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 979082 T3311 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979082 T3311 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 979082 T3311 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979083 T3311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68b475 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b3b2e2),segFN=segments_1,generation=1}
   [junit4]   2> 979083 T3311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979084 T3311 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979084 T3311 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979085 T3311 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979085 T3311 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979087 T3311 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979090 T3311 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979091 T3311 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979091 T3311 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979092 T3311 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68b475 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b3b2e2),segFN=segments_1,generation=1}
   [junit4]   2> 979092 T3311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979093 T3311 oass.SolrIndexSearcher.<init> Opening Searcher@11a8402 main
   [junit4]   2> 979093 T3312 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@11a8402 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979093 T3311 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 979094 T3301 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 979094 T3301 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 979098 T3301 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 979099 T3301 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 979100 T3314 oasc.OneIndexer.run Starting indexing thread: 3314
   [junit4]   2> 979100 T3317 oasc.OneIndexer.run Starting indexing thread: 3317
   [junit4]   2> 979100 T3318 oasc.OneIndexer.run Starting indexing thread: 3318
   [junit4]   2> 979100 T3316 oasc.OneIndexer.run Starting indexing thread: 3316
   [junit4]   2> 979100 T3315 oasc.OneIndexer.run Starting indexing thread: 3315
   [junit4]   2> 979102 T3320 oasc.OneQuery.run Starting query thread: 3320
   [junit4]   2> 979102 T3323 oasc.OneQuery.run Starting query thread: 3323
   [junit4]   2> 979102 T3322 oasc.OneQuery.run Starting query thread: 3322
   [junit4]   2> 979102 T3319 oasc.OneQuery.run Starting query thread: 3319
   [junit4]   2> 979102 T3321 oasc.OneQuery.run Starting query thread: 3321
   [junit4]   2> 979105 T3309 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 979105 T3309 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 979106 T3308 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 979106 T3308 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 979104 T3307 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 979107 T3307 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 979108 T3324 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 979108 T3324 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 979180 T3309 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979187 T3307 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979190 T3308 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979195 T3324 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 979251 T3307 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979251 T3307 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979253 T3308 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979253 T3308 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979256 T3307 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 979257 T3308 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 979269 T3309 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979270 T3309 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979278 T3309 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 979280 T3307 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979282 T3308 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979282 T3307 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979282 T3307 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 979283 T3307 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 979283 T3308 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979283 T3308 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 979284 T3308 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 979283 T3307 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 979284 T3307 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 979284 T3308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 979284 T3307 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979285 T3308 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 979285 T3307 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 979285 T3308 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979286 T3307 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979286 T3308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 979287 T3307 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fcf4d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@32cd83),segFN=segments_1,generation=1}
   [junit4]   2> 979287 T3307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979287 T3308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979288 T3307 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979288 T3308 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@49f554 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a103d),segFN=segments_1,generation=1}
   [junit4]   2> 979289 T3308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979288 T3307 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979289 T3324 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 979289 T3308 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979290 T3324 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 979289 T3307 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979290 T3308 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979291 T3307 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979291 T3308 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979292 T3308 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979292 T3307 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979299 T3308 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979301 T3307 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979301 T3307 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979302 T3307 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979302 T3324 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 979303 T3307 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fcf4d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@32cd83),segFN=segments_1,generation=1}
   [junit4]   2> 979303 T3307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979303 T3308 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979304 T3307 oass.SolrIndexSearcher.<init> Opening Searcher@3817f6 main
   [junit4]   2> 979304 T3308 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979304 T3330 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@3817f6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979305 T3308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979305 T3307 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 979305 T3308 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@49f554 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a103d),segFN=segments_1,generation=1}
   [junit4]   2> 979306 T3308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979306 T3307 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 979306 T3308 oass.SolrIndexSearcher.<init> Opening Searcher@1688381 main
   [junit4]   2> 979307 T3308 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 979308 T3308 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 979310 T3309 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979312 T3309 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979313 T3309 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 979313 T3309 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> ASYNC  NEW_CORE C9051 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979314 T3327 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979314 T3309 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 979314 T3328 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979316 T3325 C9051 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 979315 T3331 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@1688381 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979315 T3309 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 979317 T3310 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C9052 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979317 T3308 C9052 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 2
   [junit4]   2> 979318 T3309 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979319 T3309 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 979320 T3309 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979321 T3325 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 1
   [junit4]   2> 979321 T3309 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@28c42e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f33d10),segFN=segments_1,generation=1}
   [junit4]   2> 979322 T3309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979322 T3329 C9052 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979323 T3307 C9051 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 1
   [junit4]   2> 979323 T3309 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979324 T3309 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979324 T3332 C9052 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 1
   [junit4]   2> 979324 T3309 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979325 T3309 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979326 T3309 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979333 T3309 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979334 T3309 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979334 T3309 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979335 T3309 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@28c42e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f33d10),segFN=segments_1,generation=1}
   [junit4]   2> 979335 T3309 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> ASYNC  NEW_CORE C9053 name=00000_core org.apache.solr.core.SolrCore@1a99b03
   [junit4]   2> 979335 T3310 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 979335 T3309 oass.SolrIndexSearcher.<init> Opening Searcher@1cc8469 main
   [junit4]   2> 979336 T3309 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 979336 T3335 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@1cc8469 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979337 T3309 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 979339 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9054 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979342 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9055 name=00001_core org.apache.solr.core.SolrCore@1e04c58
   [junit4]   2> 979343 T3309 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 1
   [junit4]   2> 979343 T3324 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 979344 T3325 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9056 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979345 T3307 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 979345 T3324 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 979346 T3324 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 979346 T3310 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 0
   [junit4]   2> 979346 T3324 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 979347 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> 979348 T3324 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 979348 T3324 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 979349 T3324 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 979349 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 979349 T3324 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 979349 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 979350 T3324 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=38.7353515625, floorSegmentMB=0.5478515625, forceMergeDeletesPctAllowed=3.419108521801312, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 979351 T3324 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@16a1bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1641fb2),segFN=segments_1,generation=1}
   [junit4]   2> 979352 T3324 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979351 T3325 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 0
   [junit4]   2> 979353 T3324 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 979354 T3324 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 979354 T3324 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 979355 T3310 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 979355 T3324 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 979357 T3324 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 979357 T3308 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> 979360 T3329 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 979360 T3324 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 979361 T3324 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 979361 T3324 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=66.2705078125, floorSegmentMB=0.9541015625, forceMergeDeletesPctAllowed=15.859393355045421, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6816302597289294
   [junit4]   2> 979362 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 0
   [junit4]   2> 979362 T3324 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@16a1bbe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1641fb2),segFN=segments_1,generation=1}
   [junit4]   2> 979363 T3324 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 979363 T3324 oass.SolrIndexSearcher.<init> Opening Searcher@1f172e0 main
   [junit4]   2> 979364 T3324 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 979364 T3325 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 0
   [junit4]   2> 979364 T3338 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@1f172e0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 979365 T3324 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> ASYNC  NEW_CORE C9057 name=00003_core org.apache.solr.core.SolrCore@11dde9e
   [junit4]   2> 979373 T3326 C9057 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979376 T3328 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 4
   [junit4]   2> 979377 T3310 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 2
   [junit4]   2> 979376 T3324 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 1
   [junit4]   2> 979376 T3307 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 979377 T3332 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 0
   [junit4]   2> 979379 T3308 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 979380 T3309 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> 979380 T3325 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 979380 T3329 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 979381 T3326 C9055 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 979382 T3328 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 1
   [junit4]   2> 979382 T3310 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 0
   [junit4]   2> 979383 T3324 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 979383 T3307 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> 979384 T3332 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> 979384 T3308 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 0
   [junit4]   2> 979385 T3309 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 979385 T3325 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 0
   [junit4]   2> 979386 T3308 C9053 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 979386 T3326 C9054 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 0
   [junit4]   2> 979387 T3328 C9056 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 979387 T3310 C9057 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9058 name=00000_core org.apache.solr.core.SolrCore@1a99b03
   [junit4]   2> 979393 T3324 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 6
   [junit4]   2> ASYNC  NEW_CORE C9059 name=00001_core org.apache.solr.core.SolrCore@1e04c58
   [junit4]   2> 979394 T3329 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C9060 name=00004_core org.apache.solr.core.SolrCore@ca974b
   [junit4]   2> 979393 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9061 name=00002_core org.apache.solr.core.SolrCore@bce20b
   [junit4]   2> 979394 T3310 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 0
   [junit4]   2> 979395 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 979396 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 979396 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 0
   [junit4]   2> 979397 T3326 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 979397 T3328 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 979397 T3332 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C9062 name=00003_core org.apache.solr.core.SolrCore@11dde9e
   [junit4]   2> 979398 T3324 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 979399 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 979400 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 1
   [junit4]   2> 979399 T3307 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 0
   [junit4]   2> 979400 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 979401 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 979402 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 979403 T3326 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 979403 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 979404 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 979404 T3326 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 0
   [junit4]   2> 979406 T3329 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 979406 T3307 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 0
   [junit4]   2> 979407 T3310 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 979406 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 0
   [junit4]   2> 979407 T3329 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 0
   [junit4]   2> 979408 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 0
   [junit4]   2> 979409 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 979410 T3328 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 1
   [junit4]   2> 979410 T3325 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 0
   [junit4]   2> 979411 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 979412 T3325 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 979412 T3328 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 979413 T3307 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 979413 T3332 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 979413 T3329 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 0
   [junit4]   2> 979414 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 979414 T3326 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 0
   [junit4]   2> 979415 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 979415 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 979416 T3309 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 979416 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 0
   [junit4]   2> 979417 T3325 C9062 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979417 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 979418 T3328 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 979418 T3326 C9061 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979419 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 979419 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 979418 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 0
   [junit4]   2> 979420 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 979419 T3310 C9059 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979421 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 979422 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 979422 T3328 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 0
   [junit4]   2> 979424 T3308 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 979424 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 0
   [junit4]   2> 979423 T3325 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 2
   [junit4]   2> 979423 T3326 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 979424 T3332 C9062 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 979426 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 979427 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 979427 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 979427 T3324 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 979428 T3308 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 979429 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 979430 T3308 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 979430 T3329 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 0
   [junit4]   2> 979430 T3328 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 0
   [junit4]   2> 979431 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 0
   [junit4]   2> 979431 T3325 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 0
   [junit4]   2> 979433 T3310 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 979433 T3309 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 1
   [junit4]   2> 979434 T3324 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 979434 T3326 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 0
   [junit4]   2> 979434 T3332 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 0
   [junit4]   2> 979436 T3308 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> 979436 T3329 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 979438 T3325 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 979437 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 0
   [junit4]   2> 979437 T3328 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 1
   [junit4]   2> 979439 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 979440 T3324 C9061 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 979441 T3326 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 979441 T3310 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 979443 T3329 C9062 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> 979442 T3307 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 0
   [junit4]   2> 979442 T3308 C9058 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 1
   [junit4]   2> 979441 T3332 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 0
   [junit4]   2> 979447 T3325 C9060 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 1
   [junit4]   2> 979447 T3309 C9059 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 979447 T3328 C9062 oasup.LogUpdateProcessor.fini

[...truncated too long message...]

]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=3371, name=commitScheduler-2450-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=3370, name=commitScheduler-2448-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=3334, name=commitScheduler-2424-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=3337, name=commitScheduler-2429-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=3333, name=commitScheduler-2426-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=3339, name=commitScheduler-2432-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=3363, name=commitScheduler-2441-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([881BA4DF02EC43A5]:0)
   [junit4] Completed on J0 in 54.40s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 500 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:423: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:403: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1272: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:908: There were test failures: 336 suites, 1474 tests, 2 suite-level errors, 584 ignored (5 assumptions)

Total time: 45 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure