You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/14 21:21:35 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8219 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8219/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=3236, name=commitScheduler-1860-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=3195, name=commitScheduler-1831-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=3203, name=commitScheduler-1842-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=3233, name=commitScheduler-1857-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=3231, name=commitScheduler-1853-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=3234, name=commitScheduler-1848-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=3201, name=commitScheduler-1839-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=3200, name=commitScheduler-1837-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=3230, name=commitScheduler-1855-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=3199, name=commitScheduler-1835-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=3236, name=commitScheduler-1860-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=3195, name=commitScheduler-1831-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=3203, name=commitScheduler-1842-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=3233, name=commitScheduler-1857-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=3231, name=commitScheduler-1853-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=3234, name=commitScheduler-1848-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=3201, name=commitScheduler-1839-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=3200, name=commitScheduler-1837-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=3230, name=commitScheduler-1855-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=3199, name=commitScheduler-1835-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([55DEA5EF9E2E723C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3236, name=commitScheduler-1860-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=3195, name=commitScheduler-1831-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=3203, name=commitScheduler-1842-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=3233, name=commitScheduler-1857-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=3231, name=commitScheduler-1853-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=3234, name=commitScheduler-1848-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=3201, name=commitScheduler-1839-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=3200, name=commitScheduler-1837-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=3230, name=commitScheduler-1855-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=3199, name=commitScheduler-1835-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3236, name=commitScheduler-1860-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=3195, name=commitScheduler-1831-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=3203, name=commitScheduler-1842-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=3233, name=commitScheduler-1857-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=3231, name=commitScheduler-1853-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=3234, name=commitScheduler-1848-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=3201, name=commitScheduler-1839-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=3200, name=commitScheduler-1837-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=3230, name=commitScheduler-1855-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=3199, name=commitScheduler-1835-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([55DEA5EF9E2E723C]:0)




Build Log:
[...truncated 9766 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.02s J0 | OpenCloseCoreStressTest.test1HourNew
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly)
   [junit4] IGNOR/A 0.00s J0 | OpenCloseCoreStressTest.test10MinutesOld
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4]   2> 897078 T3167 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsNew
   [junit4]   2> 897087 T3167 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 897089 T3167 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40568
   [junit4]   2> 897089 T3167 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 897090 T3167 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 897090 T3167 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 897090 T3167 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/'
   [junit4]   2> 897100 T3167 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 897131 T3167 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 897131 T3167 oasc.CoreContainer.<init> New CoreContainer 1901075861
   [junit4]   2> 897132 T3167 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/]
   [junit4]   2> 897133 T3167 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 897133 T3167 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 897133 T3167 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 897134 T3167 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 897134 T3167 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 897134 T3167 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 897134 T3167 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 897135 T3167 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 897135 T3167 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 897136 T3167 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 897136 T3167 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 897137 T3167 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 897139 T3167 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_
   [junit4]   2> 897140 T3167 oasc.CorePropertiesLocator.discoverUnder Found core 00001_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/
   [junit4]   2> 897140 T3167 oasc.CorePropertiesLocator.discoverUnder Found core 00000_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/
   [junit4]   2> 897141 T3167 oasc.CorePropertiesLocator.discoverUnder Found core 00004_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/
   [junit4]   2> 897141 T3167 oasc.CorePropertiesLocator.discoverUnder Found core 00002_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/
   [junit4]   2> 897142 T3167 oasc.CorePropertiesLocator.discoverUnder Found core 00003_core in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/
   [junit4]   2> 897142 T3167 oasc.CorePropertiesLocator.discover Found 5 core definitions
   [junit4]   2> 897143 T3167 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 897143 T3167 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 897146 T3167 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 897147 T3167 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 897149 T3178 oasc.OneIndexer.run Starting indexing thread: 3178
   [junit4]   2> 897149 T3179 oasc.OneIndexer.run Starting indexing thread: 3179
   [junit4]   2> 897149 T3182 oasc.OneIndexer.run Starting indexing thread: 3182
   [junit4]   2> 897149 T3181 oasc.OneIndexer.run Starting indexing thread: 3181
   [junit4]   2> 897149 T3180 oasc.OneIndexer.run Starting indexing thread: 3180
   [junit4]   2> 897168 T3173 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 897169 T3183 oasc.OneQuery.run Starting query thread: 3183
   [junit4]   2> 897169 T3173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 897169 T3176 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 897169 T3187 oasc.OneQuery.run Starting query thread: 3187
   [junit4]   2> 897168 T3175 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 897171 T3175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 897168 T3174 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 897168 T3184 oasc.OneQuery.run Starting query thread: 3184
   [junit4]   2> 897168 T3186 oasc.OneQuery.run Starting query thread: 3186
   [junit4]   2> 897168 T3185 oasc.OneQuery.run Starting query thread: 3185
   [junit4]   2> 897173 T3174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 897170 T3176 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 897213 T3174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 897227 T3176 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 897228 T3173 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 897233 T3175 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 897255 T3174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 897255 T3174 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 897259 T3174 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 897265 T3176 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 897266 T3173 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 897266 T3176 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 897266 T3175 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 897266 T3173 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 897266 T3175 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 897269 T3176 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 897270 T3173 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 897271 T3175 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 897280 T3174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 897281 T3174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 897281 T3174 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 897282 T3174 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 897282 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 897288 T3174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 897288 T3174 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 897288 T3174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 897289 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=30.376953125, floorSegmentMB=1.0419921875, forceMergeDeletesPctAllowed=20.218378378893064, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31242134433678137
   [junit4]   2> 897289 T3174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4df845ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@44a6f6cf),segFN=segments_1,generation=1}
   [junit4]   2> 897290 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897290 T3174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 897291 T3174 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 897291 T3174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 897291 T3174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 897293 T3174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 897294 T3176 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 897295 T3176 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 897295 T3174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 897296 T3174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 897295 T3176 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 897296 T3174 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1879636043, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 897296 T3176 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 897297 T3174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4df845ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@44a6f6cf),segFN=segments_1,generation=1}
   [junit4]   2> 897297 T3174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897297 T3173 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 897297 T3176 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 897297 T3174 oass.SolrIndexSearcher.<init> Opening Searcher@692c2e7 main
   [junit4]   2> 897298 T3175 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 897298 T3176 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 897299 T3194 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@692c2e7 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 897299 T3174 oasc.SolrCores.putTransientCore Opening transient core 00000_core
   [junit4]   2> 897299 T3173 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 897300 T3174 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 897300 T3175 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 897299 T3176 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 897300 T3175 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 897301 T3175 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 897300 T3173 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 897302 T3173 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 897301 T3176 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 897302 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 897302 T3173 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 897302 T3175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 897303 T3173 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 897302 T3176 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=30.376953125, floorSegmentMB=1.0419921875, forceMergeDeletesPctAllowed=20.218378378893064, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31242134433678137
   [junit4]   2> 897303 T3173 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 897303 T3175 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 897304 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=30.376953125, floorSegmentMB=1.0419921875, forceMergeDeletesPctAllowed=20.218378378893064, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31242134433678137
   [junit4]   2> 897304 T3176 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2590de39 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43c38d4),segFN=segments_1,generation=1}
   [junit4]   2> 897305 T3173 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b5d0305 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3107fbd3),segFN=segments_1,generation=1}
   [junit4]   2> 897305 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897305 T3176 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> ASYNC  NEW_CORE C1318 name=00000_core org.apache.solr.core.SolrCore@205cca1c
   [junit4]   2> 897306 T3193 C1318 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 897306 T3176 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 897304 T3175 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 897306 T3176 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 897306 T3173 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 897307 T3176 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 897307 T3174 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 2
   [junit4]   2> 897307 T3175 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 897307 T3176 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 897307 T3173 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 897308 T3175 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=30.376953125, floorSegmentMB=1.0419921875, forceMergeDeletesPctAllowed=20.218378378893064, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31242134433678137
   [junit4]   2> 897309 T3173 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 897309 T3176 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 897309 T3175 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@75564a75 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@57f3dd39),segFN=segments_1,generation=1}
   [junit4]   2> 897309 T3173 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 897310 T3175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897310 T3174 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 0
   [junit4]   2> 897311 T3173 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 897311 T3176 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 897311 T3175 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 897312 T3176 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 897312 T3175 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 897314 T3176 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1879636043, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 897314 T3173 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 897314 T3176 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2590de39 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@43c38d4),segFN=segments_1,generation=1}
   [junit4]   2> 897315 T3176 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897314 T3173 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 897314 T3175 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 897316 T3173 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1879636043, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 897316 T3176 oass.SolrIndexSearcher.<init> Opening Searcher@780c5b5a main
   [junit4]   2> 897317 T3173 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b5d0305 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3107fbd3),segFN=segments_1,generation=1}
   [junit4]   2> 897317 T3173 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897318 T3173 oass.SolrIndexSearcher.<init> Opening Searcher@7376508a main
   [junit4]   2> 897316 T3175 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 897318 T3173 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 897317 T3196 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@780c5b5a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 897317 T3176 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 897319 T3176 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 897319 T3175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 897318 T3173 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 897318 T3197 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@7376508a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 897321 T3175 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 897322 T3175 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 897322 T3175 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1879636043, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 897323 T3175 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@75564a75 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@57f3dd39),segFN=segments_1,generation=1}
   [junit4]   2> 897323 T3175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897324 T3175 oass.SolrIndexSearcher.<init> Opening Searcher@4ac907c8 main
   [junit4]   2> 897324 T3175 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 897325 T3175 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> ASYNC  NEW_CORE C1319 name=00001_core org.apache.solr.core.SolrCore@1d68a329
   [junit4]   2> 897326 T3190 C1319 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897326 T3198 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@4ac907c8 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C1320 name=00002_core org.apache.solr.core.SolrCore@223c67dc
   [junit4]   2> 897326 T3176 C1320 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 1
   [junit4]   2> 897326 T3192 C1319 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897326 T3191 C1319 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897328 T3188 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 2
   [junit4]   2> 897327 T3173 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C1321 name=00003_core org.apache.solr.core.SolrCore@364987f1
   [junit4]   2> 897330 T3189 C1321 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897332 T3176 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 0
   [junit4]   2> 897332 T3192 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 897333 T3175 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 2
   [junit4]   2> 897334 T3174 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 897335 T3191 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> 897335 T3188 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> 897336 T3173 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 897335 T3190 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 1
   [junit4]   2> 897337 T3189 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 1
   [junit4]   2> 897338 T3192 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 897338 T3192 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 897338 T3176 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> 897339 T3174 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 1
   [junit4]   2> 897339 T3191 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 0
   [junit4]   2> 897341 T3173 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 0
   [junit4]   2> 897342 T3190 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> 897344 T3189 C1319 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 897344 T3176 C1321 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 0
   [junit4]   2> 897347 T3174 C1320 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 1
   [junit4]   2> 897347 T3191 C1318 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 1
   [junit4]   2> 897350 T3190 C1320 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 897369 T3192 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 897398 T3192 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 897399 T3192 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 897402 T3192 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 897408 T3176 C1320 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897415 T3192 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 897416 T3192 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 897416 T3192 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 897416 T3192 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 897417 T3192 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 897417 T3192 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 897417 T3192 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 897418 T3192 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 897418 T3192 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=30.376953125, floorSegmentMB=1.0419921875, forceMergeDeletesPctAllowed=20.218378378893064, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31242134433678137
   [junit4]   2> 897419 T3192 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fd9b2fa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7078b2ca),segFN=segments_1,generation=1}
   [junit4]   2> 897419 T3192 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897420 T3192 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 897420 T3192 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 897420 T3192 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 897421 T3192 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 897422 T3192 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 897424 T3192 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 897424 T3192 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 897424 T3192 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1879636043, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 897425 T3192 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@fd9b2fa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7078b2ca),segFN=segments_1,generation=1}
   [junit4]   2> 897425 T3192 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 897426 T3192 oass.SolrIndexSearcher.<init> Opening Searcher@249c10f6 main
   [junit4]   2> 897426 T3192 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 897426 T3202 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@249c10f6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 897426 T3192 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 897430 T3191 C1320 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897431 T3190 C1319 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C1322 name=00004_core org.apache.solr.core.SolrCore@71f74ade
   [junit4]   2> 897432 T3192 C1322 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> 897433 T3176 C1320 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897436 T3173 C1322 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 4
   [junit4]   2> ASYNC  NEW_CORE C1323 name=00004_core org.apache.solr.core.SolrCore@71f74ade
   [junit4]   2> 897443 T3188 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 11
   [junit4]   2> 897443 T3174 C1323 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 897443 T3189 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 10
   [junit4]   2> ASYNC  NEW_CORE C1324 name=00002_core org.apache.solr.core.SolrCore@223c67dc
   [junit4]   2> 897444 T3191 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 897444 T3175 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 0
   [junit4]   2> 897446 T3192 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C1325 name=00003_core org.apache.solr.core.SolrCore@364987f1
   [junit4]   2> 897446 T3190 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C1326 name=00001_core org.apache.solr.core.SolrCore@1d68a329
   [junit4]   2> 897447 T3176 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 1
   [junit4]   2> 897447 T3173 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 897447 T3190 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 0
   [junit4]   2> 897448 T3174 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C1327 name=00000_core org.apache.solr.core.SolrCore@205cca1c
   [junit4]   2> 897449 T3189 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 897449 T3191 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 0
   [junit4]   2> 897450 T3175 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 897451 T3188 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 897451 T3192 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 897452 T3176 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 897452 T3173 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 1
   [junit4]   2> 897453 T3190 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 897453 T3174 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 897453 T3189 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 0
   [junit4]   2> 897454 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> 897455 T3175 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 897455 T3188 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 0
   [junit4]   2> 897456 T3192 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 897456 T3176 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 0
   [junit4]   2> 897457 T3173 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 0
   [junit4]   2> 897458 T3192 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 897458 T3174 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 0
   [junit4]   2> 897459 T3189 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 897459 T3191 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 897460 T3175 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 897461 T3190 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 897461 T3188 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 1
   [junit4]   2> 897461 T3176 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 897461 T3173 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 0
   [junit4]   2> 897462 T3192 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 0
   [junit4]   2> 897463 T3174 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 897463 T3189 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 0
   [junit4]   2> 897464 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 897464 T3175 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 897465 T3190 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 897465 T3188 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 0
   [junit4]   2> 897466 T3176 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 897466 T3173 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 897467 T3176 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 897467 T3174 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 1
   [junit4]   2> 897468 T3189 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 897468 T3191 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 0
   [junit4]   2> 897468 T3175 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 0
   [junit4]   2> 897469 T3190 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 897469 T3188 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 0
   [junit4]   2> 897470 T3192 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 0
   [junit4]   2> 897471 T3173 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 897471 T3176 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 897472 T3174 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 897472 T3189 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 897473 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 1
   [junit4]   2> 897473 T3175 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 897474 T3190 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 897474 T3188 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 0
   [junit4]   2> 897475 T3192 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 897475 T3176 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 0
   [junit4]   2> 897476 T3173 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 897477 T3174 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 897477 T3189 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 897477 T3191 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 0
   [junit4]   2> 897478 T3175 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 897479 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 897479 T3190 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 0
   [junit4]   2> 897480 T3192 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 897480 T3176 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 0
   [junit4]   2> 897481 T3173 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 897481 T3174 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 0
   [junit4]   2> 897482 T3189 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 897482 T3188 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 0
   [junit4]   2> 897483 T3175 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 897483 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 897484 T3190 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 897484 T3192 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 1
   [junit4]   2> 897485 T3176 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 897485 T3173 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 0
   [junit4]   2> 897486 T3174 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 897486 T3189 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 1
   [junit4]   2> 897486 T3188 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 0
   [junit4]   2> 897487 T3175 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 0
   [junit4]   2> 897488 T3191 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 897488 T3190 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 0
   [junit4]   2> 897488 T3192 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 0
   [junit4]   2> 897489 T3176 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 897489 T3173 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 0
   [junit4]   2> 897490 T3174 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 0
   [junit4]   2> 897490 T3188 C1327 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 0
   [junit4]   2> 897491 T3191 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 897491 T3190 C1325 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 0
   [junit4]   2> 897492 T3192 C1326 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 897493 T3176 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 897493 T3189 C1323 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 897494 T3173 C1324 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 897494 T3175 C1324 oasup.LogUpdateProce

[...truncated too long message...]

:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    3) Thread[id=3203, name=commitScheduler-1842-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    4) Thread[id=3233, name=commitScheduler-1857-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    5) Thread[id=3231, name=commitScheduler-1853-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    6) Thread[id=3234, name=commitScheduler-1848-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    7) Thread[id=3201, name=commitScheduler-1839-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    8) Thread[id=3200, name=commitScheduler-1837-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    9) Thread[id=3230, name=commitScheduler-1855-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >   10) Thread[id=3199, name=commitScheduler-1835-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([55DEA5EF9E2E723C]:0)
   [junit4] Completed on J0 in 54.17s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 532 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 2 suite-level errors, 656 ignored (5 assumptions)

Total time: 42 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_45) - Build # 8224 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8224/
Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)    2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node3 in shard/collection : shard2/delLiveColl available replicas are core_node4

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node3 in shard/collection : shard2/delLiveColl available replicas are core_node4
	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092:C3D941247EC7A0AE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:154)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:141)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10954 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1241421 T2872 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /mdkl/v
   [junit4]   2> 1241424 T2872 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-DeleteReplicaTest-1384484419524
   [junit4]   2> 1241425 T2872 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1241425 T2873 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1241525 T2872 oasc.ZkTestServer.run start zk server on port:38268
   [junit4]   2> 1241526 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241530 T2879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76b842 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241530 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241531 T2872 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1241533 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241534 T2881 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f5e2 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241534 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241535 T2872 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1241537 T2872 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1241539 T2872 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1241540 T2872 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1241542 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1241542 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1241546 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1241546 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1241548 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1241549 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1241550 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1241551 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1241553 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1241553 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1241555 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1241555 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1241557 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1241557 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1241559 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1241560 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1241562 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1241563 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1241565 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1241566 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1241568 T2872 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1241568 T2872 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1241628 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241632 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41326
   [junit4]   2> 1241633 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669
   [junit4]   2> 1241633 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/'
   [junit4]   2> 1241654 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/solr.xml
   [junit4]   2> 1241691 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1241692 T2872 oasc.CoreContainer.<init> New CoreContainer 8223867
   [junit4]   2> 1241692 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/]
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1241693 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241694 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241695 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241696 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241696 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241697 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241697 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1241697 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241698 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241700 T2892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e31e5 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241700 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241702 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241703 T2894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70b955 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241703 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241705 T2872 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1241707 T2872 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1241709 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1241710 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1241711 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1241713 T2872 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1241716 T2872 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1241719 T2872 oasc.Overseer.start Overseer (id=90733570918055939-127.0.0.1:41326_mdkl%2Fv-n_0000000000) starting
   [junit4]   2> 1241724 T2872 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1241727 T2896 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1241727 T2872 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1241729 T2872 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1241730 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1241732 T2895 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1241734 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669
   [junit4]   2> 1241735 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/conf/
   [junit4]   2> 1241761 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1241762 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1241763 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1241764 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241765 T2899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e56831 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241766 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241767 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1241768 T2872 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1241827 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241828 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59732
   [junit4]   2> 1241829 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241829 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241829 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868
   [junit4]   2> 1241830 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/'
   [junit4]   2> 1241842 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/solr.xml
   [junit4]   2> 1241889 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1241890 T2872 oasc.CoreContainer.<init> New CoreContainer 27120754
   [junit4]   2> 1241890 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/]
   [junit4]   2> 1241891 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241892 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241893 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241894 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241895 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241895 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241895 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1241896 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241896 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241898 T2910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1700669 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241898 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241900 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241901 T2912 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec4066 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241901 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241907 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1242912 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1242913 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1242916 T2894 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242916 T2899 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242916 T2912 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1242920 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868
   [junit4]   2> 1242921 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/conf/
   [junit4]   2> 1242941 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1242941 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1242941 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1243000 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1243002 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49670
   [junit4]   2> 1243003 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1243003 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1243003 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041
   [junit4]   2> 1243004 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/'
   [junit4]   2> 1243015 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/solr.xml
   [junit4]   2> 1243052 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1243052 T2872 oasc.CoreContainer.<init> New CoreContainer 27567373
   [junit4]   2> 1243053 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/]
   [junit4]   2> 1243053 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1243054 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1243055 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1243056 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1243057 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1243057 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1243057 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1243058 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1243058 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1243059 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243061 T2924 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bc240 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243061 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243063 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243064 T2926 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbb468 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243064 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243067 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1244069 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1244071 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1244074 T2899 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2894 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2926 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244074 T2912 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1244085 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041
   [junit4]   2> 1244086 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/conf/
   [junit4]   2> 1244107 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1244107 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1244108 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1244172 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1244174 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42492
   [junit4]   2> 1244174 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208
   [junit4]   2> 1244175 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/'
   [junit4]   2> 1244187 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/solr.xml
   [junit4]   2> 1244223 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1244223 T2872 oasc.CoreContainer.<init> New CoreContainer 25600933
   [junit4]   2> 1244224 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/]
   [junit4]   2> 1244224 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1244225 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1244226 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1244227 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1244228 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1244228 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1244228 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1244229 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1244229 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1244230 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1244231 T2938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@628c4c name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1244231 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1244233 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1244234 T2940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c520e name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1244235 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1244238 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1245241 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42492_mdkl%2Fv
   [junit4]   2> 1245242 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42492_mdkl%2Fv
   [junit4]   2> 1245245 T2899 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2926 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2940 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2912 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245246 T2894 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1245250 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208
   [junit4]   2> 1245251 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1384484422208/conf/
   [junit4]   2> 1245276 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1245277 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1245277 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1245342 T2872 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1245344 T2872 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39331
   [junit4]   2> 1245345 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1245345 T2872 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1245345 T2872 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377
   [junit4]   2> 1245346 T2872 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/'
   [junit4]   2> 1245357 T2872 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/solr.xml
   [junit4]   2> 1245399 T2872 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1245400 T2872 oasc.CoreContainer.<init> New CoreContainer 17942655
   [junit4]   2> 1245400 T2872 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/]
   [junit4]   2> 1245401 T2872 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1245402 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1245403 T2872 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1245404 T2872 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1245405 T2872 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1245405 T2872 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1245405 T2872 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1245405 T2872 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38268/solr
   [junit4]   2> 1245406 T2872 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1245406 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1245408 T2952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e9e887 name:ZooKeeperConnection Watcher:127.0.0.1:38268 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1245408 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1245410 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1245411 T2954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ee27a name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1245411 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1245414 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246417 T2872 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1246418 T2872 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1246422 T2926 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246422 T2940 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2899 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2954 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246423 T2894 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246422 T2912 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1246427 T2872 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377
   [junit4]   2> 1246427 T2872 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/conf/
   [junit4]   2> 1246454 T2872 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1246454 T2872 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1246455 T2872 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1246456 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1246457 T2957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a1bb1d name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1246458 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1246459 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246460 T2872 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1246461 T2959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c267e2 name:ZooKeeperConnection Watcher:127.0.0.1:38268/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1246461 T2872 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1246462 T2872 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1246465 T2933 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 1246466 T2894 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1246466 T2894 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1246466 T2896 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1246467 T2896 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1247741 T2895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1247741 T2895 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1247741 T2895 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1247745 T2954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247745 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1247770 T2896 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1247770 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:59732_mdkl%2Fv
   [junit4]   2> 1247771 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:39331_mdkl%2Fv
   [junit4]   2> 1247771 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:49670_mdkl%2Fv
   [junit4]   2> 1247772 T2896 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:41326_mdkl%2Fv
   [junit4]   2> 1247773 T2908 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 1247773 T2947 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 1247773 T2920 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 1247774 T2908 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247774 T2947 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247774 T2889 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 1247774 T2920 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1247776 T2889 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1249254 T2895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1249255 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59732/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59732_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249257 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39331/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39331_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249259 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49670/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49670_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249262 T2895 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41326/mdkl/v",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41326_mdkl%2Fv",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1249264 T2940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249265 T2894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2912 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2926 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249264 T2959 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1249775 T2908 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1249775 T2947 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1249775 T2920 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1249776 T2908 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1
   [junit4]   2> 1249776 T2920 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1
   [junit4]   2> 1249777 T2920 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249776 T2889 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1249776 T2947 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2
   [junit4]   2> 1249777 T2947 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249777 T2920 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249776 T2908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249778 T2947 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249778 T2920 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249778 T2889 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/delLiveColl_shard2_replica2
   [junit4]   2> 1249779 T2908 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249779 T2947 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249779 T2908 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249779 T2889 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1249780 T2920 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249780 T2947 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249780 T2889 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1249780 T2920 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249781 T2889 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1249781 T2908 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249781 T2947 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249782 T2908 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249782 T2889 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1249783 T2889 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1249783 T2920 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249784 T2908 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249784 T2947 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249794 T2889 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1249794 T2920 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/'
   [junit4]   2> 1249795 T2908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/'
   [junit4]   2> 1249796 T2889 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1384484419669/delLiveColl_shard2_replica2/'
   [junit4]   2> 1249797 T2947 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/'
   [junit4]   2> 1249883 T2920 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249883 T2908 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249883 T2947 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249884 T2889 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1249960 T2908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249960 T2889 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249960 T2920 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249961 T2947 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1249962 T2889 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2947 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249962 T2920 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1249968 T2908 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1249968 T2947 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1249969 T2889 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1249978 T2920 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1250817 T2908 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250818 T2908 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250818 T2908 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250824 T2908 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250826 T2908 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250838 T2947 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250838 T2947 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250839 T2947 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250845 T2947 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250845 T2920 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250845 T2908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250845 T2920 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250846 T2920 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250847 T2947 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250856 T2908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250859 T2908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250860 T2920 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250860 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250861 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250861 T2908 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250862 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250862 T2908 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250863 T2908 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250863 T2908 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/
   [junit4]   2> 1250863 T2920 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250863 T2908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250864 T2908 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data
   [junit4]   2> 1250865 T2908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 1250865 T2908 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250866 T2908 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1384484419868/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 1250866 T2947 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250866 T2908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250867 T2908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d0954e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@51ccba),segFN=segments_1,generation=1}
   [junit4]   2> 1250867 T2908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250869 T2947 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250871 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250872 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250872 T2947 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250873 T2908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250873 T2908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250873 T2908 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250873 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250874 T2908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250874 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250874 T2920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250875 T2947 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250875 T2908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250876 T2908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250876 T2908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250876 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250885 T2947 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250885 T2947 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250886 T2947 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/
   [junit4]   2> 1250886 T2920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250886 T2908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250886 T2947 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250888 T2947 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data
   [junit4]   2> 1250888 T2947 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 1250888 T2947 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250889 T2947 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1384484423377/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 1250889 T2908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250889 T2947 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250889 T2920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250889 T2908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250890 T2947 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10273f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1293525),segFN=segments_1,generation=1}
   [junit4]   2> 1250890 T2908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250890 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250891 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250890 T2947 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250891 T2920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250891 T2908 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@d0954e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@51ccba),segFN=segments_1,generation=1}
   [junit4]   2> 1250892 T2889 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250892 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250893 T2920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250892 T2908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250893 T2920 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1250892 T2889 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1250894 T2920 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/
   [junit4]   2> 1250893 T2908 oass.SolrIndexSearcher.<init> Opening Searcher@b687b3 main
   [junit4]   2> 1250894 T2889 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1250894 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250894 T2920 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e8c82f
   [junit4]   2> 1250895 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250896 T2920 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data
   [junit4]   2> 1250896 T2920 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 1250896 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250896 T2920 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1250897 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250897 T2920 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1384484421041/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 1250897 T2964 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@b687b3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250898 T2920 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.33875771996404735]
   [junit4]   2> 1250897 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250898 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250898 T2920 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@11cbdad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105b1cf),segFN=segments_1,generation=1}
   [junit4]   2> 1250899 T2920 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250899 T2908 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 1250899 T2908 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:59732/mdkl/v collection:delLiveColl shard:shard1
   [junit4]   2> 1250899 T2947 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250900 T2947 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250900 T2947 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250900 T2908 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 1250900 T2947 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250901 T2947 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250901 T2947 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250902 T2947 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250902 T2889 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1250902 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1250902 T2947 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250903 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250903 T2947 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250903 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1250904 T2947 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250904 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1250905 T2920 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250905 T2920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1250906 T2908 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1250906 T2889 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1250906 T2920 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1250907 T2920 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1250907 T2947 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250908 T2920 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1250908 T2908 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 1250908 T2947 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250908 T2920 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1250909 T2920 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1250909 T2947 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250909 T2920 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1250910 T2947 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10273f9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1293525),segFN=segments_1,generation=1}
   [junit4]   2> 1250910 T2920 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1250910 T2947 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250911 T2947 oass.SolrIndexSearcher.<init> Opening Searcher@3481b0 main
   [junit4]   2> 1250911 T2920 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1250914 T2920 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1250914 T2920 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1250914 T2965 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@3481b0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250923 T2920 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=46.333984375, floorSegmentMB=1.44140625, forceMergeDeletesPctAllowed=1.8643109108065081, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1250923 T2920 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@11cbdad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105b1cf),segFN=segments_1,generation=1}
   [junit4]   2> 1250924 T2920 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1250924 T2947 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 1250924 T2920 oass.SolrIndexSearcher.<init> Opening Searcher@1db64d3 main
   [junit4]   2> 1250924 T2947 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:39331/mdkl/v collection:delLiveColl shard:shard1
   [junit4]   2> 1250926 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250926 T2966 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@1db64d3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1250928 T2920 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 1250928 T2920 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:49670/mdkl/v collection:delLiveColl shard:shard2
   [junit4]   2> 1250929 T2920 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 1250929 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1250932 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250934 T2920 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1250934 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1250935 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1250936 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1250936 T2889 oas

[...truncated too long message...]

r.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1347737 T2958 oaz.ClientCnxn$SendThread.run WARN Session 0x14259b3b7c6000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 1348237 T2958 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1348737 T2958 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1348738 T2958 oaz.ClientCnxn$SendThread.run WARN Session 0x14259b3b7c6000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 107818 T2871 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ar_KW, timezone=Europe/Skopje
   [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.7.0_45 (32-bit)/cpus=8,threads=3,free=110151640,total=265453568
   [junit4]   2> NOTE: All tests run in this JVM: [TestSweetSpotSimilarityFactory, HighlighterTest, PingRequestHandlerTest, TestSolr4Spatial, ShardRoutingCustomTest, TestAnalyzedSuggestions, TermVectorComponentDistributedTest, TestCollapseQParserPlugin, MinimalSchemaTest, TestRandomFaceting, PluginInfoTest, RAMDirectoryFactoryTest, HdfsUnloadDistributedZkTest, DistanceFunctionTest, DirectSolrConnectionTest, DocValuesMultiTest, PrimUtilsTest, TestSolrQueryParserDefaultOperatorResource, SolrPluginUtilsTest, TestUniqueKeyFieldResource, TestRangeQuery, TestDynamicFieldCollectionResource, TestSolrXml, TestUtils, TestQuerySenderNoQuery, TestManagedSchemaFieldResource, TestNumberUtils, HdfsSyncSliceTest, TestFastWriter, TestQuerySenderListener, RequestHandlersTest, TestFastLRUCache, TestFoldingMultitermQuery, TestZkChroot, TestFastOutputStream, MultiTermTest, TimeZoneUtilsTest, SolrTestCaseJ4Test, SolrIndexConfigTest, SpellingQueryConverterTest, DateFieldTest, ChaosMonkeySafeLeaderTest, CurrencyFieldOpenExchangeTest, HdfsDirectoryTest, RequiredFieldsTest, TestQueryTypes, TestCloudManagedSchemaAddField, SolrRequestParserTest, TestWordDelimiterFilterFactory, OpenExchangeRatesOrgProviderTest, TestLazyCores, UniqFieldsUpdateProcessorFactoryTest, DirectSolrSpellCheckerTest, TestRecoveryHdfs, TestCSVLoader, HighlighterConfigTest, InfoHandlerTest, BasicDistributedZk2Test, SortByFunctionTest, TestDefaultSimilarityFactory, XmlUpdateRequestHandlerTest, BadIndexSchemaTest, SpellCheckComponentTest, TestAtomicUpdateErrorCases, BlockCacheTest, ZkNodePropsTest, XsltUpdateRequestHandlerTest, HdfsChaosMonkeySafeLeaderTest, CoreAdminCreateDiscoverTest, TestSolrJ, DeleteInactiveReplicaTest, TestPhraseSuggestions, PreAnalyzedFieldTest, QueryResultKeyTest, FileUtilsTest, NotRequiredUniqueKeyTest, TestRTGBase, TestRecovery, HardAutoCommitTest, IndexBasedSpellCheckerTest, DefaultValueUpdateProcessorTest, TestJmxIntegration, SyncSliceTest, LeaderElectionIntegrationTest, BasicZkTest, LeaderElectionTest, TestRandomDVFaceting, TestStressReorder, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, ConvertedLegacyTest, TestFiltering, SpellCheckCollatorTest, TestStressLucene, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, DebugComponentTest, StandardRequestHandlerTest, TestOmitPositions, TermsComponentTest, TestValueSourceCache, TestIndexingPerformance, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, TestCSVResponseWriter, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, UpdateParamsTest, TestStressRecovery, AssignTest, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=423FCF3C0998C092 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Europe/Skopje -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    >    2) Thread[id=2959, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=2958, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[423FCF3C0998C092]-SendThread(localhost.localdomain:38268), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([423FCF3C0998C092]:0)
   [junit4] Completed on J1 in 107.86s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 127 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 3 suite-level errors, 1 error, 230 ignored (6 assumptions)

Total time: 52 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8223 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8223/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2031, name=commitScheduler-1674-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=2033, name=commitScheduler-1678-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)




Build Log:
[...truncated 9598 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.01s J1 | OpenCloseCoreStressTest.test10MinutesOld
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4]   2> 857219 T1995 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 857229 T1995 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 857235 T1995 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34247
   [junit4]   2> 857235 T1995 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 857236 T1995 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857236 T1995 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 857237 T1995 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/'
   [junit4]   2> 857249 T1995 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 857297 T1995 oasc.CoreContainer.<init> New CoreContainer 1985104213
   [junit4]   2> 857298 T1995 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/]
   [junit4]   2> 857299 T1995 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 857299 T1995 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 857300 T1995 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 857300 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 857301 T1995 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 857302 T1995 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 857302 T1995 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 857303 T1995 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 857304 T1995 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 857304 T1995 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 857332 T2005 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 857333 T2005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 857379 T2005 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857423 T2005 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857423 T2005 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857427 T2005 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 857439 T2005 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857441 T2005 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857441 T2005 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 857441 T2005 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 857442 T2005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 857442 T2005 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 857442 T2005 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857442 T2005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 857443 T2005 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857443 T2005 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6f5d7f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ac81960),segFN=segments_1,generation=1}
   [junit4]   2> 857444 T2005 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857445 T2005 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857445 T2005 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857447 T2005 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857449 T2005 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857449 T2005 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857450 T2005 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857451 T2005 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6f5d7f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ac81960),segFN=segments_1,generation=1}
   [junit4]   2> 857467 T2005 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857468 T2005 oass.SolrIndexSearcher.<init> Opening Searcher@2c8e5e85 main
   [junit4]   2> 857468 T2005 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 857469 T2006 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@2c8e5e85 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857469 T1995 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 857470 T1995 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 857473 T1995 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 857474 T1995 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 857476 T2009 oasc.OneIndexer.run Starting indexing thread: 2009
   [junit4]   2> 857476 T2012 oasc.OneIndexer.run Starting indexing thread: 2012
   [junit4]   2> 857476 T2011 oasc.OneIndexer.run Starting indexing thread: 2011
   [junit4]   2> 857476 T2010 oasc.OneIndexer.run Starting indexing thread: 2010
   [junit4]   2> 857476 T2008 oasc.OneIndexer.run Starting indexing thread: 2008
   [junit4]   2> 857479 T2013 oasc.OneQuery.run Starting query thread: 2013
   [junit4]   2> 857480 T2002 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 857480 T2014 oasc.OneQuery.run Starting query thread: 2014
   [junit4]   2> 857480 T2017 oasc.OneQuery.run Starting query thread: 2017
   [junit4]   2> 857480 T2016 oasc.OneQuery.run Starting query thread: 2016
   [junit4]   2> 857480 T2015 oasc.OneQuery.run Starting query thread: 2015
   [junit4]   2> 857481 T2002 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 857480 T2004 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 857480 T2003 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 857483 T2004 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 857482 T2018 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 857483 T2003 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> 857484 T2018 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> ASYNC  NEW_CORE C6576 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857487 T2001 C6576 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 2
   [junit4]   2> 857551 T2018 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857554 T2003 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857565 T2002 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> ASYNC  NEW_CORE C6577 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857609 T2023 C6577 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=15 
   [junit4]   2> 857629 T2004 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 857666 T2018 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857667 T2018 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857669 T2003 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857670 T2003 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857682 T2018 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 857683 T2003 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 857688 T2002 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857689 T2002 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857703 T2002 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 857734 T2004 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857734 T2018 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857735 T2004 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 857736 T2018 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857736 T2018 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 857736 T2003 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857736 T2018 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 857738 T2018 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 857738 T2003 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857739 T2018 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 857739 T2003 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 857740 T2003 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 857741 T2004 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 857739 T2018 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857743 T2002 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857741 T2003 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 857758 T2003 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 857759 T2003 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857758 T2018 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 857760 T2003 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 857759 T2002 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857761 T2003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857760 T2018 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857762 T2003 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8ff4ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c624362),segFN=segments_1,generation=1}
   [junit4]   2> 857762 T2018 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@487164a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c42455),segFN=segments_1,generation=1}
   [junit4]   2> 857763 T2018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857761 T2002 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 857764 T2002 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 857764 T2018 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857762 T2003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857765 T2002 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 857765 T2002 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 857765 T2004 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 857764 T2018 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857766 T2002 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857766 T2003 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857767 T2002 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 857767 T2018 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857768 T2002 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857767 T2003 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857767 T2004 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857769 T2002 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@354523c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@166ea673),segFN=segments_1,generation=1}
   [junit4]   2> 857770 T2002 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857769 T2003 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857768 T2018 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857771 T2002 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857771 T2003 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857769 T2004 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 857772 T2018 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857772 T2004 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 857772 T2003 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857771 T2002 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857773 T2004 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 857773 T2002 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857773 T2004 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 857774 T2002 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857774 T2018 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857775 T2003 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857774 T2004 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857776 T2002 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 857776 T2004 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 857776 T2003 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857775 T2018 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857777 T2003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857777 T2004 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=42, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 857778 T2003 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1c8ff4ae lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c624362),segFN=segments_1,generation=1}
   [junit4]   2> 857779 T2002 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 857779 T2002 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 857777 T2018 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857780 T2002 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857779 T2004 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2930dd1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2baea459),segFN=segments_1,generation=1}
   [junit4]   2> 857781 T2004 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857779 T2003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857781 T2002 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@354523c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@166ea673),segFN=segments_1,generation=1}
   [junit4]   2> 857782 T2002 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857780 T2018 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@487164a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c42455),segFN=segments_1,generation=1}
   [junit4]   2> 857783 T2018 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857782 T2002 oass.SolrIndexSearcher.<init> Opening Searcher@64598f1 main
   [junit4]   2> 857782 T2004 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857782 T2003 oass.SolrIndexSearcher.<init> Opening Searcher@20272646 main
   [junit4]   2> 857784 T2004 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 857785 T2002 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 857783 T2018 oass.SolrIndexSearcher.<init> Opening Searcher@1674c73e main
   [junit4]   2> 857786 T2002 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 857787 T2018 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 857785 T2004 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 857785 T2027 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@20272646 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857785 T2003 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 857785 T2028 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@64598f1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857800 T2003 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> ASYNC  NEW_CORE C6578 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857800 T2023 C6578 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857796 T2004 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 857787 T2018 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 857787 T2026 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@1674c73e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857803 T2004 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> ASYNC  NEW_CORE C6579 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857801 T2002 C6579 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 1
   [junit4]   2> 857800 T2025 C6579 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857813 T2004 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> ASYNC  NEW_CORE C6580 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857815 T2003 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 1
   [junit4]   2> 857815 T2020 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 2
   [junit4]   2> 857817 T2022 C6580 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857817 T2021 C6580 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857817 T2002 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 1
   [junit4]   2> 857816 T2004 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> ASYNC  NEW_CORE C6581 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857818 T2024 C6581 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857819 T2004 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 857820 T2025 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 857819 T2018 C6581 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 1
   [junit4]   2> 857819 T2023 C6580 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 0
   [junit4]   2> 857821 T2003 C6579 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 857820 T2004 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2930dd1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2baea459),segFN=segments_1,generation=1}
   [junit4]   2> 857822 T2004 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 857822 T2004 oass.SolrIndexSearcher.<init> Opening Searcher@5cf5b025 main
   [junit4]   2> 857823 T2004 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 857823 T2022 C6581 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> 857823 T2030 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@5cf5b025 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 857823 T2004 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> ASYNC  NEW_CORE C6582 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857834 T2024 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6583 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857839 T2025 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C6584 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857839 T2004 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 1
   [junit4]   2> 857839 T2021 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 2
   [junit4]   2> 857840 T2002 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 4
   [junit4]   2> 857840 T2020 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 1
   [junit4]   2> 857842 T2020 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 857843 T2003 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6585 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857842 T2023 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 0
   [junit4]   2> 857844 T2024 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 857843 T2022 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 857846 T2025 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6586 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857846 T2004 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 857847 T2002 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 0
   [junit4]   2> 857847 T2021 C6583 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 1
   [junit4]   2> 857848 T2018 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 1
   [junit4]   2> 857849 T2021 C6584 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 0
   [junit4]   2> 857849 T2020 C6582 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 857850 T2023 C6586 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 0
   [junit4]   2> 857851 T2024 C6585 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6587 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857860 T2022 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 9
   [junit4]   2> 857861 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6588 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857861 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 857860 T2025 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 9
   [junit4]   2> ASYNC  NEW_CORE C6589 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857864 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6590 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857864 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 857865 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 5
   [junit4]   2> ASYNC  NEW_CORE C6591 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857865 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 857867 T2024 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 1
   [junit4]   2> 857867 T2004 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 0
   [junit4]   2> 857866 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 857869 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 1
   [junit4]   2> 857870 T2025 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 857869 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 0
   [junit4]   2> 857870 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 0
   [junit4]   2> 857870 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 857873 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 1
   [junit4]   2> 857873 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 857874 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 857874 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 857874 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 857876 T2003 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 857876 T2004 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 1
   [junit4]   2> 857877 T2002 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 1
   [junit4]   2> 857878 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 857878 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 857879 T2021 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 857881 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 2
   [junit4]   2> 857881 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 857881 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 1
   [junit4]   2> 857883 T2003 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 1
   [junit4]   2> 857883 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 857884 T2004 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 857885 T2022 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 857885 T2002 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 1
   [junit4]   2> 857886 T2025 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 857886 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 0
   [junit4]   2> 857888 T2024 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 857888 T2021 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 857888 T2020 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 857890 T2003 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 857890 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 857892 T2004 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 857893 T2022 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 857894 T2002 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 857894 T2018 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 857895 T2022 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 857895 T2002 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 857896 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 857898 T2018 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 857899 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 857900 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 857901 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 857903 T2018 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 857904 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 857904 T2021 C6589 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857905 T2023 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 857905 T2025 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 857906 T2021 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 857906 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 857907 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 857908 T2020 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 857909 T2018 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 857909 T2024 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 857910 T2004 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 857911 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 857912 T2025 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 857912 T2021 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 857912 T2002 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 3
   [junit4]   2> 857913 T2003 C6591 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 857914 T2022 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 857915 T2020 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 857916 T2003 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 2
   [junit4]   2> 857917 T2022 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 2
   [junit4]   2> 857917 T2023 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 857917 T2025 C6588 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857916 T2024 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 857919 T2002 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 857920 T2021 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 857920 T2004 C6591 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857920 T2018 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 857921 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 1
   [junit4]   2> 857922 T2003 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 857921 T2020 C6587 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 857922 T2023 C6588 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 857923 T2025 C6587 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 857925 T2022 C6589 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 2
   [junit4]   2> 857925 T2004 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 1
   [junit4]   2> 857925 T2021 C6590 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 857925 T2023 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 2
   [junit4]   2> 857927 T2024 C6588 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6592 name=00000_core org.apache.solr.core.SolrCore@93f21ef
   [junit4]   2> 857937 T2020 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 10
   [junit4]   2> ASYNC  NEW_CORE C6593 name=00001_core org.apache.solr.core.SolrCore@59f485c2
   [junit4]   2> 857937 T2003 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 10
   [junit4]   2> ASYNC  NEW_CORE C6594 name=00002_core org.apache.solr.core.SolrCore@74160aa8
   [junit4]   2> 857938 T2002 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6595 name=00003_core org.apache.solr.core.SolrCore@2bca32d4
   [junit4]   2> 857938 T2025 C6595 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 857939 T2018 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 857941 T2022 C6594 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 857942 T2004 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 857942 T2022 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 1
   [junit4]   2> 857942 T2023 C6593 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 857942 T2024 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 0
   [junit4]   2> 857944 T2020 C6592 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C6596 name=00004_core org.apache.solr.core.SolrCore@79397428
   [junit4]   2> 857945 T2003 C6596 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2}

[...truncated too long message...]


   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    3) Thread[id=2061, name=commitScheduler-1690-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    4) Thread[id=2065, name=commitScheduler-1696-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    5) Thread[id=2019, name=commitScheduler-1666-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    6) Thread[id=2063, name=commitScheduler-1693-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    7) Thread[id=2032, name=commitScheduler-1673-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    8) Thread[id=2059, name=commitScheduler-1687-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    9) Thread[id=2057, name=commitScheduler-1685-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >   10) Thread[id=2029, name=commitScheduler-1676-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8DFFD4DC74EE63F7]:0)
   [junit4] Completed on J1 in 54.40s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 600 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 2 suite-level errors, 842 ignored (5 assumptions)

Total time: 41 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0) - Build # 8222 - Still Failing!

Posted by Michael McCandless <lu...@mikemccandless.com>.
I already committed a fix for this.

Mike McCandless

http://blog.mikemccandless.com


On Thu, Nov 14, 2013 at 6:56 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8222/
> Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
>
> 1 tests failed.
> REGRESSION:  org.apache.lucene.index.TestIndexWriterReader.testTooManySegments
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
>         at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
>         at org.junit.Assert.fail(Assert.java:92)
>         at org.junit.Assert.assertTrue(Assert.java:43)
>         at org.junit.Assert.assertTrue(Assert.java:54)
>         at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>         at java.lang.Thread.run(Thread.java:662)
>
>
>
>
> Build Log:
> [...truncated 256 lines...]
>    [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterReader -Dtests.method=testTooManySegments -Dtests.seed=92DE792121089547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=UTF-8
>    [junit4] FAILURE 6.42s J0 | TestIndexWriterReader.testTooManySegments <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
>    [junit4]    >        at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:662)
>    [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=mk, timezone=Europe/Amsterdam
>    [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.6.0_45 (32-bit)/cpus=8,threads=1,free=28777776,total=80531456
>    [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterReader]
>    [junit4] Completed on J0 in 25.62s, 23 tests, 1 failure <<< FAILURES!
>
> [...truncated 1321 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 379 suites, 2758 tests, 1 failure, 105 ignored (92 assumptions)
>
> Total time: 24 minutes 21 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-4.x-Linux (32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0) - Build # 8222 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8222/
Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt

1 tests failed.
REGRESSION:  org.apache.lucene.index.TestIndexWriterReader.testTooManySegments

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:774)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:683)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:44)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 256 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterReader
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterReader -Dtests.method=testTooManySegments -Dtests.seed=92DE792121089547 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Europe/Amsterdam -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 6.42s J0 | TestIndexWriterReader.testTooManySegments <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([92DE792121089547:D44DB20BF8EF2D94]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterReader.testTooManySegments(TestIndexWriterReader.java:1149)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> NOTE: test params are: codec=Appending, sim=DefaultSimilarity, locale=mk, timezone=Europe/Amsterdam
   [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.6.0_45 (32-bit)/cpus=8,threads=1,free=28777776,total=80531456
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexWriterReader]
   [junit4] Completed on J0 in 25.62s, 23 tests, 1 failure <<< FAILURES!

[...truncated 1321 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:49: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 379 suites, 2758 tests, 1 failure, 105 ignored (92 assumptions)

Total time: 24 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jrockit-jdk1.6.0_45-R28.2.7-4.1.0 -XnoOpt
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8221 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8221/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=890, name=commitScheduler-756-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=930, name=commitScheduler-777-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=936, name=commitScheduler-786-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=904, name=commitScheduler-768-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=932, name=commitScheduler-780-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=899, name=commitScheduler-760-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=934, name=commitScheduler-783-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=927, name=commitScheduler-774-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=900, name=commitScheduler-763-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=902, name=commitScheduler-765-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=890, name=commitScheduler-756-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=930, name=commitScheduler-777-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=936, name=commitScheduler-786-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=904, name=commitScheduler-768-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=932, name=commitScheduler-780-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=899, name=commitScheduler-760-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=934, name=commitScheduler-783-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=927, name=commitScheduler-774-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=900, name=commitScheduler-763-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=902, name=commitScheduler-765-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([C2CF8A9245FCD80C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=890, name=commitScheduler-756-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    2) Thread[id=930, name=commitScheduler-777-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    3) Thread[id=936, name=commitScheduler-786-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    4) Thread[id=904, name=commitScheduler-768-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    5) Thread[id=932, name=commitScheduler-780-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    6) Thread[id=899, name=commitScheduler-760-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    7) Thread[id=934, name=commitScheduler-783-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    8) Thread[id=927, name=commitScheduler-774-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)    9) Thread[id=900, name=commitScheduler-763-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)   10) Thread[id=902, name=commitScheduler-765-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)         at java.lang.Thread.run(Thread.java:662)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=890, name=commitScheduler-756-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   2) Thread[id=930, name=commitScheduler-777-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   3) Thread[id=936, name=commitScheduler-786-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   4) Thread[id=904, name=commitScheduler-768-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   5) Thread[id=932, name=commitScheduler-780-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   6) Thread[id=899, name=commitScheduler-760-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   7) Thread[id=934, name=commitScheduler-783-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   8) Thread[id=927, name=commitScheduler-774-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
   9) Thread[id=900, name=commitScheduler-763-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
  10) Thread[id=902, name=commitScheduler-765-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
        at java.lang.Thread.run(Thread.java:662)
	at __randomizedtesting.SeedInfo.seed([C2CF8A9245FCD80C]:0)




Build Log:
[...truncated 9475 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> 297388 T867 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 297398 T867 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 297401 T867 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45942
   [junit4]   2> 297402 T867 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 297402 T867 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 297402 T867 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 297403 T867 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/'
   [junit4]   2> 297414 T867 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 297455 T867 oasc.CoreContainer.<init> New CoreContainer 2134311424
   [junit4]   2> 297456 T867 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/]
   [junit4]   2> 297457 T867 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 297457 T867 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 297458 T867 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 297458 T867 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 297458 T867 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 297458 T867 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 297459 T867 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 297459 T867 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 297459 T867 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 297461 T867 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 297461 T867 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 297461 T867 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 297487 T877 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 297488 T877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 297530 T877 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 297578 T877 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 297578 T877 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 297581 T877 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 297595 T877 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297596 T877 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297596 T877 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 297596 T877 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 297597 T877 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 297597 T877 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 297597 T877 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 297598 T877 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 297598 T877 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.860644174886858]
   [junit4]   2> 297599 T877 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@332320b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ac70f3b),segFN=segments_1,generation=1}
   [junit4]   2> 297599 T877 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297600 T877 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 297600 T877 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 297601 T877 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 297601 T877 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 297603 T877 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 297604 T877 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 297605 T877 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 297605 T877 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18239518873696856]
   [junit4]   2> 297605 T877 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@332320b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ac70f3b),segFN=segments_1,generation=1}
   [junit4]   2> 297606 T877 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297606 T877 oass.SolrIndexSearcher.<init> Opening Searcher@49ca5775 main
   [junit4]   2> 297606 T878 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@49ca5775 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 297606 T877 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 297608 T867 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 297608 T867 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 297621 T867 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 297622 T867 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 297625 T880 oasc.OneIndexer.run Starting indexing thread: 880
   [junit4]   2> 297625 T883 oasc.OneIndexer.run Starting indexing thread: 883
   [junit4]   2> 297625 T881 oasc.OneIndexer.run Starting indexing thread: 881
   [junit4]   2> 297625 T884 oasc.OneIndexer.run Starting indexing thread: 884
   [junit4]   2> 297625 T882 oasc.OneIndexer.run Starting indexing thread: 882
   [junit4]   2> 297632 T886 oasc.OneQuery.run Starting query thread: 886
   [junit4]   2> 297632 T885 oasc.OneQuery.run Starting query thread: 885
   [junit4]   2> 297633 T889 oasc.OneQuery.run Starting query thread: 889
   [junit4]   2> 297632 T888 oasc.OneQuery.run Starting query thread: 888
   [junit4]   2> 297632 T887 oasc.OneQuery.run Starting query thread: 887
   [junit4]   2> 297634 T876 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 297634 T875 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 297636 T875 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 297635 T876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 297636 T874 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 297637 T874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> ASYNC  NEW_CORE C2508 name=00000_core org.apache.solr.core.SolrCore@75fe6111
   [junit4]   2> 297645 T873 C2508 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 4
   [junit4]   2> 297693 T875 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 297700 T874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 297703 T876 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 297738 T894 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 297738 T894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> ASYNC  NEW_CORE C2509 name=00000_core org.apache.solr.core.SolrCore@75fe6111
   [junit4]   2> 297739 T893 C2509 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 297741 T896 C2509 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297759 T875 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 297759 T874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 297759 T875 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 297759 T874 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 297763 T875 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 297763 T874 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 297771 T876 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 297772 T876 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 297777 T876 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 297787 T875 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297793 T875 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297793 T874 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297793 T875 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 297794 T875 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 297795 T875 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 297795 T874 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297795 T875 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 297796 T874 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 297796 T874 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 297796 T875 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 297797 T874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 297798 T875 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 297798 T874 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> 297798 T894 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 297798 T875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.860644174886858]
   [junit4]   2> 297798 T874 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 297799 T875 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7da6143 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e49838f),segFN=segments_1,generation=1}
   [junit4]   2> 297800 T875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297800 T874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 297801 T874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.860644174886858]
   [junit4]   2> 297801 T875 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 297802 T875 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 297802 T874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ae2ce62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@417763a3),segFN=segments_1,generation=1}
   [junit4]   2> 297802 T875 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 297806 T874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297806 T875 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 297807 T874 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 297808 T875 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 297808 T874 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 297809 T874 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 297809 T874 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 297810 T876 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297810 T875 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 297810 T875 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 297811 T875 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18239518873696856]
   [junit4]   2> 297811 T874 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 297811 T876 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297811 T875 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7da6143 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e49838f),segFN=segments_1,generation=1}
   [junit4]   2> 297812 T875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297812 T876 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 297812 T875 oass.SolrIndexSearcher.<init> Opening Searcher@65186833 main
   [junit4]   2> 297813 T876 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 297813 T874 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 297814 T874 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 297814 T897 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@65186833 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 297814 T875 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 297814 T874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18239518873696856]
   [junit4]   2> 297814 T876 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 297816 T874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ae2ce62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@417763a3),segFN=segments_1,generation=1}
   [junit4]   2> 297816 T874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297816 T876 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 297815 T875 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 297827 T876 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 297827 T874 oass.SolrIndexSearcher.<init> Opening Searcher@ed0abba main
   [junit4]   2> 297828 T876 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 297829 T874 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 297830 T898 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@ed0abba main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 297829 T876 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.860644174886858]
   [junit4]   2> 297830 T874 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> 297831 T876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68c8af42 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7f37c6cd),segFN=segments_1,generation=1}
   [junit4]   2> 297832 T876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> ASYNC  NEW_CORE C2510 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 297833 T895 C2510 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297842 T876 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 297845 T876 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 297845 T876 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 297845 T876 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> ASYNC  NEW_CORE C2511 name=00001_core org.apache.solr.core.SolrCore@38089d5
   [junit4]   2> 297846 T892 C2511 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297847 T876 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 297848 T896 C2511 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C2512 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 297848 T891 C2512 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 15
   [junit4]   2> 297847 T875 C2512 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 14
   [junit4]   2> 297851 T873 C2511 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 3
   [junit4]   2> 297851 T876 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 297857 T876 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> ASYNC  NEW_CORE C2513 name=00001_core org.apache.solr.core.SolrCore@38089d5
   [junit4]   2> 297857 T895 C2513 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297857 T876 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18239518873696856]
   [junit4]   2> 297859 T876 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@68c8af42 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7f37c6cd),segFN=segments_1,generation=1}
   [junit4]   2> 297859 T891 C2513 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 2
   [junit4]   2> 297859 T876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297859 T874 C2513 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 8
   [junit4]   2> 297860 T876 oass.SolrIndexSearcher.<init> Opening Searcher@27225261 main
   [junit4]   2> 297861 T876 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 297861 T901 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@27225261 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 297861 T876 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> 297873 T894 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> ASYNC  NEW_CORE C2514 name=00004_core org.apache.solr.core.SolrCore@355259ca
   [junit4]   2> 297873 T896 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 2
   [junit4]   2> 297873 T892 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 2
   [junit4]   2> 297873 T873 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 2
   [junit4]   2> 297874 T894 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 297874 T876 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 3
   [junit4]   2> 297876 T875 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 1
   [junit4]   2> 297877 T895 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2515 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 297877 T891 C2515 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 297878 T874 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 2
   [junit4]   2> 297878 T896 C2515 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> 297879 T892 C2515 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 0
   [junit4]   2> 297879 T894 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 297880 T873 C2515 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 297880 T876 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 1
   [junit4]   2> 297881 T895 C2515 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 297881 T891 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 0
   [junit4]   2> 297882 T874 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 0
   [junit4]   2> 297883 T896 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 297883 T892 C2514 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C2516 name=00004_core org.apache.solr.core.SolrCore@355259ca
   [junit4]   2> 297889 T873 C2516 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 6
   [junit4]   2> ASYNC  NEW_CORE C2517 name=00001_core org.apache.solr.core.SolrCore@38089d5
   [junit4]   2> 297890 T895 C2517 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2518 name=00000_core org.apache.solr.core.SolrCore@75fe6111
   [junit4]   2> 297890 T876 C2518 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 1
   [junit4]   2> 297891 T891 C2517 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 297893 T896 C2517 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 1
   [junit4]   2> 297893 T892 C2517 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 297893 T873 C2516 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 0
   [junit4]   2> 297896 T876 C2517 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 297896 T891 C2518 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 0
   [junit4]   2> 297899 T892 C2518 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 297899 T896 C2516 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 1
   [junit4]   2> 297899 T894 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 297901 T894 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 297901 T894 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 297901 T894 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 297902 T894 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 297902 T894 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 297902 T894 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> ASYNC  NEW_CORE C2519 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 297902 T873 C2519 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 297903 T894 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 297903 T894 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.860644174886858]
   [junit4]   2> 297904 T894 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2d68a498 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17414937),segFN=segments_1,generation=1}
   [junit4]   2> 297904 T894 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297905 T894 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 297905 T891 C2518 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 0
   [junit4]   2> 297906 T894 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 297906 T894 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 297906 T894 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 297907 T894 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 297908 T892 C2518 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 297909 T894 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 297910 T894 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 297910 T894 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=12, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18239518873696856]
   [junit4]   2> 297910 T894 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2d68a498 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17414937),segFN=segments_1,generation=1}
   [junit4]   2> 297911 T894 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 297911 T894 oass.SolrIndexSearcher.<init> Opening Searcher@4c9e5e43 main
   [junit4]   2> 297912 T894 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 297912 T903 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@4c9e5e43 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 297912 T894 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> ASYNC  NEW_CORE C2520 name=00003_core org.apache.solr.core.SolrCore@251814
   [junit4]   2> 297923 T875 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 2
   [junit4]   2> 297924 T895 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 2
   [junit4]   2> 297934 T894 C2520 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=11 
   [junit4]   2> 297935 T876 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 2
   [junit4]   2> 297935 T896 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 2
   [junit4]   2> 297935 T874 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2521 name=00001_core org.apache.solr.core.SolrCore@38089d5
   [junit4]   2> 297937 T873 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2522 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 297937 T891 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 1
   [junit4]   2> 297938 T892 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2523 name=00004_core org.apache.solr.core.SolrCore@355259ca
   [junit4]   2> 297938 T895 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C2524 name=00000_core org.apache.solr.core.SolrCore@75fe6111
   [junit4]   2> 297938 T875 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 1
   [junit4]   2> 297940 T894 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 297941 T894 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 1
   [junit4]   2> 297941 T876 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 297942 T894 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 297942 T876 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 0
   [junit4]   2> 297943 T873 C2524 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297944 T894 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 1
   [junit4]   2> 297944 T874 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 1
   [junit4]   2> 297944 T875 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 297945 T896 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 1
   [junit4]   2> 297946 T876 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 297948 T891 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 2
   [junit4]   2> 297948 T892 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 297948 T895 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 1
   [junit4]   2> 297949 T873 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 297951 T894 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 2
   [junit4]   2> 297952 T873 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 2
   [junit4]   2> 297952 T875 C2522 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297954 T896 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 297954 T876 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 297955 T873 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 297955 T892 C2522 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297955 T891 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 297956 T874 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 3
   [junit4]   2> 297957 T894 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 297957 T875 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 1
   [junit4]   2> 297959 T895 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 2
   [junit4]   2> 297959 T876 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 1
   [junit4]   2> 297960 T873 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 297960 T896 C2524 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 297960 T892 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 297961 T873 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 0
   [junit4]   2> 297962 T892 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 297963 T873 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 297963 T892 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 1
   [junit4]   2> 297964 T873 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 297964 T891 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 0
   [junit4]   2> 297965 T892 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 297966 T874 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 297966 T894 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 1
   [junit4]   2> 297967 T875 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 297968 T894 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 1
   [junit4]   2> 297968 T876 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 1
   [junit4]   2> 297969 T896 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> 297969 T873 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 1
   [junit4]   2> 297970 T892 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> 297971 T874 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 0
   [junit4]   2> 297972 T892 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> 297973 T874 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 297970 T891 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 0
   [junit4]   2> 297973 T875 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 297974 T876 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 1
   [junit4]   2> 297975 T873 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 297975 T896 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 0
   [junit4]   2> 297976 T895 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 297976 T892 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 297977 T894 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> 297978 T874 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 297978 T891 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 297979 T875 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 1
   [junit4]   2> 297979 T876 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 1
   [junit4]   2> 297980 T873 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 1
   [junit4]   2> 297981 T896 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 297981 T895 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 297982 T892 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 297982 T894 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 1
   [junit4]   2> 297983 T874 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 297983 T891 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 0
   [junit4]   2> 297984 T875 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 297985 T876 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 1
   [junit4]   2> 297985 T873 C2523 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 0
   [junit4]   2> 297986 T896 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 1
   [junit4]   2> 297986 T895 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 0
   [junit4]   2> 297987 T873 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 1
   [junit4]   2> 297988 T892 C2521 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 1
   [junit4]   2> 297989 T874 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 1
   [junit4]   2> 297989 T875 C2522 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 1
   [junit4]   2> 297989 T891 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 297990 T876 C2524 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 297990 T896 C2520 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C2525 name=00000_core org.apache.solr.core.SolrCore@75fe6111
   [junit4]   2> 297992 T895 C2525 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 297997 T894 C2525 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 6
   [junit4]   2> 297998 T892 C2525 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2526 name=00003_core org.apache.solr.core.SolrCore@251814
   [junit4]   2> 297998 T874 C2526 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2527 name=00004_core org.apache.solr.core.SolrCore@355259ca
   [junit4]   2> 297998 T873 C2527 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2528 name=00002_core org.apache.solr.core.SolrCore@2a8f9112
   [junit4]   2> 298000 T875 C2528 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 298000 T874 C2525 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id130]} 0 1
   [junit4]   2> 298001 T891 C2525 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/u

[...truncated too long message...]

ecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    3) Thread[id=936, name=commitScheduler-786-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    4) Thread[id=904, name=commitScheduler-768-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    5) Thread[id=932, name=commitScheduler-780-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    6) Thread[id=899, name=commitScheduler-760-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    7) Thread[id=934, name=commitScheduler-783-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    8) Thread[id=927, name=commitScheduler-774-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >    9) Thread[id=900, name=commitScheduler-763-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    >   10) Thread[id=902, name=commitScheduler-765-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917)
   [junit4]    >         at java.lang.Thread.run(Thread.java:662)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C2CF8A9245FCD80C]:0)
   [junit4] Completed on J1 in 54.50s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 797 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 2 suite-level errors, 914 ignored (4 assumptions)

Total time: 40 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_45) - Build # 8220 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8220/
Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest:     1) Thread[id=1301, name=commitScheduler-719-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=1340, name=commitScheduler-739-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=1344, name=commitScheduler-746-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=1311, name=commitScheduler-728-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1341, name=commitScheduler-743-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=1312, name=commitScheduler-726-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=1298, name=commitScheduler-716-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=1308, name=commitScheduler-722-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=1335, name=commitScheduler-734-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=1342, name=commitScheduler-741-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.core.OpenCloseCoreStressTest: 
   1) Thread[id=1301, name=commitScheduler-719-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=1340, name=commitScheduler-739-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=1344, name=commitScheduler-746-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=1311, name=commitScheduler-728-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=1341, name=commitScheduler-743-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=1312, name=commitScheduler-726-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=1298, name=commitScheduler-716-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=1308, name=commitScheduler-722-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=1335, name=commitScheduler-734-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=1342, name=commitScheduler-741-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([CBB7BE732B560CE0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1301, name=commitScheduler-719-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=1340, name=commitScheduler-739-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    3) Thread[id=1344, name=commitScheduler-746-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    4) Thread[id=1311, name=commitScheduler-728-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    5) Thread[id=1341, name=commitScheduler-743-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    6) Thread[id=1312, name=commitScheduler-726-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    7) Thread[id=1298, name=commitScheduler-716-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    8) Thread[id=1308, name=commitScheduler-722-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    9) Thread[id=1335, name=commitScheduler-734-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)   10) Thread[id=1342, name=commitScheduler-741-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1301, name=commitScheduler-719-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=1340, name=commitScheduler-739-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   3) Thread[id=1344, name=commitScheduler-746-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   4) Thread[id=1311, name=commitScheduler-728-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   5) Thread[id=1341, name=commitScheduler-743-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   6) Thread[id=1312, name=commitScheduler-726-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   7) Thread[id=1298, name=commitScheduler-716-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   8) Thread[id=1308, name=commitScheduler-722-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   9) Thread[id=1335, name=commitScheduler-734-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
  10) Thread[id=1342, name=commitScheduler-741-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([CBB7BE732B560CE0]:0)




Build Log:
[...truncated 10433 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test10MinutesNew
   [junit4]    > Assumption #1: 'nightly' test group is disabled (@Nightly)
   [junit4]   2> 579159 T1274 oas.SolrTestCaseJ4.setUp ###Starting test15SecondsOld
   [junit4]   2> 579169 T1274 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 579173 T1274 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37819
   [junit4]   2> 579173 T1274 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 579174 T1274 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 579174 T1274 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_
   [junit4]   2> 579174 T1274 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/'
   [junit4]   2> 579184 T1274 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/solr.xml
   [junit4]   2> 579217 T1274 oasc.CoreContainer.<init> New CoreContainer 488958552
   [junit4]   2> 579217 T1274 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/]
   [junit4]   2> 579218 T1274 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 579218 T1274 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 579218 T1274 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 579219 T1274 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 579219 T1274 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 579219 T1274 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 579219 T1274 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 579219 T1274 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 579220 T1274 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 579221 T1274 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 579221 T1274 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 579221 T1274 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 579241 T1284 oasc.CoreContainer.create Creating SolrCore '00000_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core
   [junit4]   2> 579242 T1284 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/'
   [junit4]   2> 579268 T1284 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 579295 T1284 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 579295 T1284 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 579298 T1284 oass.IndexSchema.readSchema [00000_core] Schema name=tiny
   [junit4]   2> 579310 T1284 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579311 T1284 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579311 T1284 oasc.SolrCore.<init> [00000_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/
   [junit4]   2> 579311 T1284 oasc.SolrCore.<init> JMX monitoring not detected for core: 00000_core
   [junit4]   2> 579312 T1284 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data
   [junit4]   2> 579312 T1284 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index/
   [junit4]   2> 579312 T1284 oasc.SolrCore.initIndex WARN [00000_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579313 T1284 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00000_core/data/index
   [junit4]   2> 579313 T1284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 579314 T1284 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a5aee48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@48ea35c2),segFN=segments_1,generation=1}
   [junit4]   2> 579314 T1284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579314 T1284 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579315 T1284 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 579315 T1284 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 579315 T1284 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 579317 T1284 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579319 T1284 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579319 T1284 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579319 T1284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 579320 T1284 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5a5aee48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@48ea35c2),segFN=segments_1,generation=1}
   [junit4]   2> 579320 T1284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579320 T1284 oass.SolrIndexSearcher.<init> Opening Searcher@37b1589 main
   [junit4]   2> 579321 T1284 oasc.CoreContainer.registerCore registering core: 00000_core
   [junit4]   2> 579321 T1285 oasc.SolrCore.registerSearcher [00000_core] Registered new searcher Searcher@37b1589 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 579321 T1274 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 579321 T1274 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 579325 T1274 oasc.OpenCloseCoreStressTest.doStress Starting indexing and querying
   [junit4]   2> 579325 T1274 oasc.OpenCloseCoreStressTest.doStress 
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	
   [junit4]   2> 	Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 579326 T1288 oasc.OneIndexer.run Starting indexing thread: 1288
   [junit4]   2> 579326 T1287 oasc.OneIndexer.run Starting indexing thread: 1287
   [junit4]   2> 579327 T1289 oasc.OneIndexer.run Starting indexing thread: 1289
   [junit4]   2> 579327 T1290 oasc.OneIndexer.run Starting indexing thread: 1290
   [junit4]   2> 579327 T1291 oasc.OneIndexer.run Starting indexing thread: 1291
   [junit4]   2> 579333 T1280 oasc.CoreContainer.create Creating SolrCore '00003_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core
   [junit4]   2> 579334 T1280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/'
   [junit4]   2> 579335 T1294 oasc.OneQuery.run Starting query thread: 1294
   [junit4]   2> 579335 T1293 oasc.OneQuery.run Starting query thread: 1293
   [junit4]   2> 579335 T1295 oasc.OneQuery.run Starting query thread: 1295
   [junit4]   2> 579335 T1296 oasc.OneQuery.run Starting query thread: 1296
   [junit4]   2> 579334 T1292 oasc.OneQuery.run Starting query thread: 1292
   [junit4]   2> 579334 T1282 oasc.CoreContainer.create Creating SolrCore '00001_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core
   [junit4]   2> 579336 T1282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/'
   [junit4]   2> ASYNC  NEW_CORE C640 name=00000_core org.apache.solr.core.SolrCore@57a3154b
   [junit4]   2> 579354 T1297 C640 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id5]} 0 4
   [junit4]   2> 579372 T1280 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 579383 T1282 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 579399 T1280 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 579399 T1280 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 579403 T1280 oass.IndexSchema.readSchema [00003_core] Schema name=tiny
   [junit4]   2> 579410 T1282 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 579410 T1282 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 579414 T1280 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579415 T1280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579415 T1280 oasc.SolrCore.<init> [00003_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/
   [junit4]   2> 579416 T1280 oasc.SolrCore.<init> JMX monitoring not detected for core: 00003_core
   [junit4]   2> 579415 T1282 oass.IndexSchema.readSchema [00001_core] Schema name=tiny
   [junit4]   2> 579416 T1280 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data
   [junit4]   2> 579416 T1280 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index/
   [junit4]   2> 579416 T1280 oasc.SolrCore.initIndex WARN [00003_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579417 T1280 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00003_core/data/index
   [junit4]   2> 579417 T1280 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 579418 T1280 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@113cce69 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@714eda05),segFN=segments_1,generation=1}
   [junit4]   2> 579418 T1280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579419 T1280 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579419 T1280 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 579419 T1280 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 579420 T1280 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 579421 T1280 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579422 T1280 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579423 T1280 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579423 T1280 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 579423 T1280 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@113cce69 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@714eda05),segFN=segments_1,generation=1}
   [junit4]   2> 579424 T1280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579424 T1280 oass.SolrIndexSearcher.<init> Opening Searcher@7ebb79b9 main
   [junit4]   2> 579425 T1280 oasc.SolrCores.putTransientCore Opening transient core 00003_core
   [junit4]   2> 579425 T1300 oasc.SolrCore.registerSearcher [00003_core] Registered new searcher Searcher@7ebb79b9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 579425 T1280 oasc.CoreContainer.registerCore registering core: 00003_core
   [junit4]   2> 579427 T1282 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579428 T1282 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579429 T1282 oasc.SolrCore.<init> [00001_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/
   [junit4]   2> 579429 T1282 oasc.SolrCore.<init> JMX monitoring not detected for core: 00001_core
   [junit4]   2> 579429 T1282 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data
   [junit4]   2> 579429 T1282 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index/
   [junit4]   2> ASYNC  NEW_CORE C641 name=00003_core org.apache.solr.core.SolrCore@37234922
   [junit4]   2> 579430 T1280 C641 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id1]} 0 2
   [junit4]   2> 579430 T1282 oasc.SolrCore.initIndex WARN [00001_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579430 T1282 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00001_core/data/index
   [junit4]   2> 579431 T1282 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 579431 T1282 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6d967375 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5e149aac),segFN=segments_1,generation=1}
   [junit4]   2> 579431 T1282 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579432 T1281 C641 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id2]} 0 1
   [junit4]   2> 579432 T1282 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579432 T1280 oasc.CoreContainer.create Creating SolrCore '00004_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core
   [junit4]   2> 579433 T1280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/'
   [junit4]   2> 579433 T1282 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 579433 T1282 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 579434 T1282 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 579435 T1282 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579438 T1304 oasc.CoreContainer.create Creating SolrCore '00002_core' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core
   [junit4]   2> 579438 T1304 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/'
   [junit4]   2> 579439 T1303 C641 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 579440 T1282 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579440 T1305 C640 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 579439 T1302 C641 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 579440 T1282 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579441 T1282 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 579442 T1282 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6d967375 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5e149aac),segFN=segments_1,generation=1}
   [junit4]   2> 579442 T1282 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579442 T1282 oass.SolrIndexSearcher.<init> Opening Searcher@421b7bc main
   [junit4]   2> 579443 T1282 oasc.SolrCores.putTransientCore Opening transient core 00001_core
   [junit4]   2> 579443 T1307 oasc.SolrCore.registerSearcher [00001_core] Registered new searcher Searcher@421b7bc main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 579443 T1282 oasc.CoreContainer.registerCore registering core: 00001_core
   [junit4]   2> ASYNC  NEW_CORE C642 name=00001_core org.apache.solr.core.SolrCore@752d8a81
   [junit4]   2> 579452 T1282 C642 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id3]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C643 name=00001_core org.apache.solr.core.SolrCore@752d8a81
   [junit4]   2> 579452 T1281 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id8]} 0 3
   [junit4]   2> 579452 T1283 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id4]} 0 3
   [junit4]   2> 579452 T1299 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id6]} 0 3
   [junit4]   2> 579461 T1282 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id12]} 0 1
   [junit4]   2> 579461 T1303 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id9]} 0 1
   [junit4]   2> 579462 T1305 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id10]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C644 name=00003_core org.apache.solr.core.SolrCore@37234922
   [junit4]   2> 579464 T1283 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id13]} 0 1
   [junit4]   2> 579465 T1281 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id14]} 0 1
   [junit4]   2> 579466 T1306 C643 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 579469 T1303 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id17]} 0 1
   [junit4]   2> 579473 T1305 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id18]} 0 1
   [junit4]   2> 579476 T1280 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 579478 T1304 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 579503 T1280 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 579503 T1280 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 579504 T1304 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 579505 T1304 oass.IndexSchema.readSchema Reading Solr Schema from schema-tiny.xml
   [junit4]   2> 579507 T1280 oass.IndexSchema.readSchema [00004_core] Schema name=tiny
   [junit4]   2> 579508 T1304 oass.IndexSchema.readSchema [00002_core] Schema name=tiny
   [junit4]   2> 579518 T1280 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579519 T1280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579519 T1280 oasc.SolrCore.<init> [00004_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/
   [junit4]   2> 579519 T1280 oasc.SolrCore.<init> JMX monitoring not detected for core: 00004_core
   [junit4]   2> 579519 T1304 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579519 T1280 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data
   [junit4]   2> 579520 T1280 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index/
   [junit4]   2> 579520 T1280 oasc.SolrCore.initIndex WARN [00004_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579520 T1304 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579520 T1280 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00004_core/data/index
   [junit4]   2> 579520 T1304 oasc.SolrCore.<init> [00002_core] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/
   [junit4]   2> 579521 T1304 oasc.SolrCore.<init> JMX monitoring not detected for core: 00002_core
   [junit4]   2> 579521 T1280 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5812462887470863]
   [junit4]   2> 579521 T1304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data
   [junit4]   2> 579521 T1304 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index/
   [junit4]   2> 579522 T1304 oasc.SolrCore.initIndex WARN [00002_core] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579522 T1280 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@611cc4df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4a72a2f3),segFN=segments_1,generation=1}
   [junit4]   2> 579522 T1304 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./OpenCloseCoreStressTest_/00002_core/data/index
   [junit4]   2> 579522 T1280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579522 T1304 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=25, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 579523 T1280 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579523 T1304 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4d48c7f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@544a43e9),segFN=segments_1,generation=1}
   [junit4]   2> 579523 T1304 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579523 T1280 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 579524 T1280 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 579524 T1304 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579524 T1280 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 579524 T1304 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 579525 T1304 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 579525 T1304 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 579525 T1280 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579526 T1304 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579527 T1280 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579527 T1280 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579527 T1304 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579527 T1280 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=9.80078125, floorSegmentMB=1.3564453125, forceMergeDeletesPctAllowed=25.625016173711657, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 579528 T1304 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579528 T1280 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@611cc4df lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4a72a2f3),segFN=segments_1,generation=1}
   [junit4]   2> 579528 T1304 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 579528 T1280 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579529 T1304 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4d48c7f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@544a43e9),segFN=segments_1,generation=1}
   [junit4]   2> 579529 T1304 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579529 T1280 oass.SolrIndexSearcher.<init> Opening Searcher@74503cbe main
   [junit4]   2> 579529 T1304 oass.SolrIndexSearcher.<init> Opening Searcher@636cee2a main
   [junit4]   2> 579530 T1309 oasc.SolrCore.registerSearcher [00004_core] Registered new searcher Searcher@74503cbe main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 579530 T1280 oasc.SolrCores.putTransientCore Opening transient core 00004_core
   [junit4]   2> 579530 T1304 oasc.SolrCores.putTransientCore Opening transient core 00002_core
   [junit4]   2> 579530 T1310 oasc.SolrCore.registerSearcher [00002_core] Registered new searcher Searcher@636cee2a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 579531 T1304 oasc.CoreContainer.registerCore registering core: 00002_core
   [junit4]   2> 579530 T1280 oasc.CoreContainer.registerCore registering core: 00004_core
   [junit4]   2> ASYNC  NEW_CORE C645 name=00002_core org.apache.solr.core.SolrCore@40fc10b7
   [junit4]   2> 579534 T1304 C645 oasc.SolrCore.execute [00002_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 579537 T1302 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id11]} 0 2
   [junit4]   2> 579537 T1299 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id15]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C646 name=00004_core org.apache.solr.core.SolrCore@73e4d8c5
   [junit4]   2> 579538 T1280 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id7]} 0 2
   [junit4]   2> 579540 T1282 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id16]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C647 name=00000_core org.apache.solr.core.SolrCore@57a3154b
   [junit4]   2> 579541 T1281 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id20]} 0 1
   [junit4]   2> 579541 T1306 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id21]} 0 1
   [junit4]   2> 579541 T1303 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id22]} 0 1
   [junit4]   2> 579541 T1283 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id19]} 0 1
   [junit4]   2> 579542 T1305 C646 oasc.SolrCore.execute [00004_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 579543 T1280 C643 oasc.SolrCore.execute [00001_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 579543 T1304 C647 oasc.SolrCore.execute [00000_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 579543 T1299 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id23]} 0 1
   [junit4]   2> 579544 T1281 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id26]} 0 0
   [junit4]   2> 579544 T1282 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id25]} 0 1
   [junit4]   2> 579544 T1302 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id24]} 0 0
   [junit4]   2> 579545 T1306 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id27]} 0 1
   [junit4]   2> 579547 T1283 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id29]} 0 1
   [junit4]   2> 579547 T1305 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id30]} 0 0
   [junit4]   2> 579547 T1303 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id28]} 0 1
   [junit4]   2> 579548 T1280 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id31]} 0 1
   [junit4]   2> 579548 T1283 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id32]} 0 1
   [junit4]   2> 579549 T1299 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id33]} 0 0
   [junit4]   2> 579550 T1281 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id34]} 0 1
   [junit4]   2> 579550 T1282 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id35]} 0 1
   [junit4]   2> 579550 T1299 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id36]} 0 0
   [junit4]   2> 579551 T1306 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id37]} 0 1
   [junit4]   2> 579552 T1305 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id38]} 0 1
   [junit4]   2> 579553 T1304 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id39]} 0 1
   [junit4]   2> 579553 T1305 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id41]} 0 1
   [junit4]   2> 579553 T1280 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id40]} 0 0
   [junit4]   2> 579553 T1283 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id42]} 0 0
   [junit4]   2> 579554 T1302 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id43]} 0 0
   [junit4]   2> 579555 T1281 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id44]} 0 0
   [junit4]   2> 579556 T1282 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id45]} 0 1
   [junit4]   2> 579556 T1281 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id46]} 0 0
   [junit4]   2> 579556 T1299 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id47]} 0 0
   [junit4]   2> 579557 T1303 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id48]} 0 1
   [junit4]   2> 579558 T1304 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id49]} 0 1
   [junit4]   2> 579558 T1305 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id50]} 0 0
   [junit4]   2> 579559 T1280 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id51]} 0 1
   [junit4]   2> 579559 T1302 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id53]} 0 0
   [junit4]   2> 579559 T1283 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id52]} 0 0
   [junit4]   2> 579560 T1306 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id54]} 0 1
   [junit4]   2> 579561 T1282 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id55]} 0 1
   [junit4]   2> 579561 T1306 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id56]} 0 0
   [junit4]   2> 579562 T1282 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id57]} 0 1
   [junit4]   2> 579562 T1303 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id58]} 0 1
   [junit4]   2> 579563 T1304 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id59]} 0 1
   [junit4]   2> 579563 T1305 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id60]} 0 0
   [junit4]   2> 579564 T1280 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id61]} 0 1
   [junit4]   2> 579564 T1302 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id62]} 0 0
   [junit4]   2> 579565 T1283 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id63]} 0 1
   [junit4]   2> 579565 T1281 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id64]} 0 1
   [junit4]   2> 579566 T1299 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id65]} 0 1
   [junit4]   2> 579566 T1306 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id66]} 0 1
   [junit4]   2> 579567 T1282 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id67]} 0 1
   [junit4]   2> 579568 T1303 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id68]} 0 1
   [junit4]   2> 579568 T1304 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id69]} 0 1
   [junit4]   2> 579568 T1302 C644 oasc.SolrCore.execute [00003_core] webapp=/solr path=/select params={q=*:*&wt=javabin&qt=/select&version=2} hits=0 status=0 QTime=0 
   [junit4]   2> 579568 T1305 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id70]} 0 0
   [junit4]   2> 579569 T1280 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id71]} 0 1
   [junit4]   2> 579569 T1281 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id72]} 0 0
   [junit4]   2> 579571 T1299 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id74]} 0 1
   [junit4]   2> 579571 T1283 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id73]} 0 1
   [junit4]   2> 579572 T1306 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id75]} 0 1
   [junit4]   2> 579572 T1299 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id77]} 0 1
   [junit4]   2> 579572 T1282 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id76]} 0 1
   [junit4]   2> 579573 T1305 C644 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id78]} 0 0
   [junit4]   2> 579574 T1280 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id79]} 0 1
   [junit4]   2> 579575 T1305 C645 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id80]} 0 1
   [junit4]   2> 579575 T1281 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id81]} 0 1
   [junit4]   2> 579575 T1283 C643 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id82]} 0 1
   [junit4]   2> 579576 T1304 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id83]} 0 0
   [junit4]   2> 579577 T1306 C646 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id84]} 0 1
   [junit4]   2> 579577 T1302 C647 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id85]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C648 name=00001_core org.apache.solr.core.SolrCore@752d8a81
   [junit4]   2> 579578 T1299 C648 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id87]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C649 name=00003_core org.apache.solr.core.SolrCore@37234922
   [junit4]   2> 579585 T1302 C649 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id89]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C650 name=00004_core org.apache.solr.core.SolrCore@73e4d8c5
   [junit4]   2> 579586 T1303 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id88]} 0 2
   [junit4]   2> 579587 T1305 C648 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id90]} 0 1
   [junit4]   2> 579587 T1282 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id86]} 0 10
   [junit4]   2> 579587 T1281 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id91]} 0 1
   [junit4]   2> 579588 T1283 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id92]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C651 name=00000_core org.apache.solr.core.SolrCore@57a3154b
   [junit4]   2> 579590 T1306 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id94]} 0 1
   [junit4]   2> 579590 T1304 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id93]} 0 1
   [junit4]   2> 579590 T1299 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id96]} 0 1
   [junit4]   2> 579590 T1280 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id95]} 0 1
   [junit4]   2> 579591 T1302 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id97]} 0 1
   [junit4]   2> 579593 T1282 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id100]} 0 1
   [junit4]   2> ASYNC  NEW_CORE C652 name=00002_core org.apache.solr.core.SolrCore@40fc10b7
   [junit4]   2> 579593 T1303 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id98]} 0 1
   [junit4]   2> 579594 T1281 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id101]} 0 1
   [junit4]   2> 579593 T1305 C648 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id99]} 0 1
   [junit4]   2> 579594 T1283 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id102]} 0 1
   [junit4]   2> 579595 T1306 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id103]} 0 0
   [junit4]   2> 579596 T1304 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id104]} 0 0
   [junit4]   2> 579597 T1280 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id106]} 0 1
   [junit4]   2> 579596 T1299 C649 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id105]} 0 0
   [junit4]   2> 579598 T1304 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id107]} 0 1
   [junit4]   2> 579598 T1299 C649 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id108]} 0 1
   [junit4]   2> 579599 T1303 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id109]} 0 1
   [junit4]   2> 579599 T1305 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id111]} 0 0
   [junit4]   2> 579600 T1281 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id110]} 0 1
   [junit4]   2> 579601 T1283 C648 oasup.LogUpdateProcessor.finish [00001_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id112]} 0 1
   [junit4]   2> 579601 T1306 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id113]} 0 1
   [junit4]   2> 579602 T1302 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id115]} 0 0
   [junit4]   2> 579603 T1280 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id114]} 0 2
   [junit4]   2> 579603 T1282 C649 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id116]} 0 1
   [junit4]   2> 579604 T1304 C649 oasup.LogUpdateProcessor.finish [00003_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id117]} 0 1
   [junit4]   2> 579604 T1299 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id118]} 0 1
   [junit4]   2> 579605 T1303 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id119]} 0 0
   [junit4]   2> 579606 T1305 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id120]} 0 1
   [junit4]   2> 579606 T1281 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id121]} 0 0
   [junit4]   2> 579607 T1283 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id122]} 0 1
   [junit4]   2> 579607 T1306 C651 oasup.LogUpdateProcessor.finish [00000_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id123]} 0 1
   [junit4]   2> 579608 T1302 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id124]} 0 1
   [junit4]   2> 579609 T1280 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id125]} 0 1
   [junit4]   2> 579610 T1282 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id126]} 0 1
   [junit4]   2> 579610 T1304 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id127]} 0 1
   [junit4]   2> 579610 T1303 C650 oasup.LogUpdateProcessor.finish [00004_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id129]} 0 1
   [junit4]   2> 579611 T1299 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={wt=javabin&version=2} {add=[id128]} 0 2
   [junit4]   2> 579611 T1305 C652 oasup.LogUpdateProcessor.finish [00002_core] webapp=/solr path=/update params={w

[...truncated too long message...]

eStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    4) Thread[id=1311, name=commitScheduler-728-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    5) Thread[id=1341, name=commitScheduler-743-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    6) Thread[id=1312, name=commitScheduler-726-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    7) Thread[id=1298, name=commitScheduler-716-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    8) Thread[id=1308, name=commitScheduler-722-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    9) Thread[id=1335, name=commitScheduler-734-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >   10) Thread[id=1342, name=commitScheduler-741-thread-1, state=WAITING, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CBB7BE732B560CE0]:0)
   [junit4] Completed on J1 in 54.34s, 6 tests, 2 errors, 4 skipped <<< FAILURES!

[...truncated 595 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:429: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:409: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:489: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:912: There were test failures: 336 suites, 1478 tests, 2 suite-level errors, 791 ignored (4 assumptions)

Total time: 32 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure