You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/08/02 05:17:32 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_51) - Build # 13696 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/13696/
Java: 32bit/jdk1.8.0_51 -server -XX:+UseConcMarkSweepGC

11 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.search.TestStressRecovery:     1) Thread[id=5893, name=WRITER6, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    2) Thread[id=5888, name=WRITER1, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    3) Thread[id=5890, name=WRITER3, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    4) Thread[id=5889, name=WRITER2, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    5) Thread[id=5892, name=WRITER5, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    6) Thread[id=5894, name=WRITER7, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    7) Thread[id=5887, name=WRITER0, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    8) Thread[id=5891, name=WRITER4, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)    9) Thread[id=5895, name=WRITER8, state=WAITING, group=TGRP-TestStressRecovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)         at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)         at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.search.TestStressRecovery: 
   1) Thread[id=5893, name=WRITER6, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   2) Thread[id=5888, name=WRITER1, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   3) Thread[id=5890, name=WRITER3, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   4) Thread[id=5889, name=WRITER2, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   5) Thread[id=5892, name=WRITER5, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   6) Thread[id=5894, name=WRITER7, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   7) Thread[id=5887, name=WRITER0, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   8) Thread[id=5891, name=WRITER4, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
   9) Thread[id=5895, name=WRITER8, state=WAITING, group=TGRP-TestStressRecovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
        at org.apache.solr.search.TestStressRecovery$1.run(TestStressRecovery.java:112)
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5901, name=READER5, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5901, name=READER5, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5898, name=READER2, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5898, name=READER2, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5902, name=READER6, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5902, name=READER6, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5897, name=READER1, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5897, name=READER1, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5899, name=READER3, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5899, name=READER3, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5896, name=READER0, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5896, name=READER0, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5903, name=READER7, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5903, name=READER7, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.TestStressRecovery

Error Message:
Captured an uncaught exception in thread: Thread[id=5900, name=READER4, state=RUNNABLE, group=TGRP-TestStressRecovery]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5900, name=READER4, state=RUNNABLE, group=TGRP-TestStressRecovery]
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090]:0)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:331)
Caused by: java.lang.NullPointerException
	at org.apache.solr.util.TestHarness.query(TestHarness.java:320)
	at org.apache.solr.util.TestHarness.query(TestHarness.java:302)
	at org.apache.solr.search.TestStressRecovery$2.run(TestStressRecovery.java:306)


FAILED:  org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090:9757E07511C2D329]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForRecoveriesToFinish(BaseCdcrDistributedZkTest.java:465)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.clearSourceCollection(BaseCdcrDistributedZkTest.java:319)
	at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTestPartialReplication(CdcrReplicationHandlerTest.java:86)
	at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest(CdcrReplicationHandlerTest.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:966)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:941)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.search.TestStressRecovery.testStressRecovery

Error Message:
java.lang.AssertionError

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([301358D17C79C090:8A29318CE3917F9E]:0)
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at org.apache.solr.search.TestStressRecovery.testStressRecovery(TestStressRecovery.java:370)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.update.CdcrUpdateLogTest$1.run(CdcrUpdateLogTest.java:444)
	at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1320)
	at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1258)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:209)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more




Build Log:
[...truncated 10175 lines...]
   [junit4] Suite: org.apache.solr.search.TestStressRecovery
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001
   [junit4]   2> 492177 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 492177 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 492177 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 492177 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 492177 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 492190 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 492203 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 492216 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.Config Loaded SolrConfig: solrconfig-tlog.xml
   [junit4]   2> 492216 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.IndexSchema Reading Solr Schema from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
   [junit4]   2> 492219 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 492291 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 492291 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 492292 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 492293 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 492297 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 492297 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 492297 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 492304 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.CoreContainer New CoreContainer 3369080
   [junit4]   2> 492304 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 492304 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.CoreContainer loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/lib
   [junit4]   2> 492304 WARN  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/lib).
   [junit4]   2> 492319 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 492322 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 492323 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 492323 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 492323 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 492323 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.c.SolrCore Created CoreDescriptor: {name=collection1, config=solrconfig-tlog.xml, transient=false, schema=schema15.xml, loadOnStartup=true, configSetProperties=configsetprops.json, instanceDir=collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001, shard=shard1}
   [junit4]   2> 492324 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 492324 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 492324 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 492334 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 492336 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 492344 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.Config Loaded SolrConfig: solrconfig-tlog.xml
   [junit4]   2> 492344 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.IndexSchema Reading Solr Schema from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
   [junit4]   2> 492348 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 492422 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 492423 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 492423 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 492425 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 492429 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf'
   [junit4]   2> 492429 INFO  (coreLoadExecutor-2425-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 492429 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 492429 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/], dataDir=[null]
   [junit4]   2> 492429 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1959570
   [junit4]   2> 492430 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001
   [junit4]   2> 492430 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001/index/
   [junit4]   2> 492430 WARN  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 492430 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001/index
   [junit4]   2> 492430 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=94.4443359375, floorSegmentMB=1.8583984375, forceMergeDeletesPctAllowed=11.683007811025542, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 492431 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_1,generation=1}
   [junit4]   2> 492431 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 492431 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.search.TestStressRecovery_301358D17C79C090-001/init-core-data-001/
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 492433 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 492434 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 492435 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 492435 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 492436 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 492437 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 492438 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 492438 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 492438 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=92.837890625, floorSegmentMB=1.38671875, forceMergeDeletesPctAllowed=15.420896548784487, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_1,generation=1}
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 492439 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@11f8752[collection1] main
   [junit4]   2> 492440 WARN  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Cannot write to config directory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 492440 INFO  (searcherExecutor-2426-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@11f8752[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 492440 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 492441 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1508359688177057792
   [junit4]   2> 492442 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.u.UpdateLog Took 2 ms to seed version buckets with highest version 1508359688177057792
   [junit4]   2> 492442 INFO  (coreLoadExecutor-2425-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 492443 INFO  (SUITE-TestStressRecovery-seed#[301358D17C79C090]-worker) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 492445 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStressRecovery
   [junit4]   2> 492445 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.c.SolrCore [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 492446 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.s.SolrIndexSearcher Opening Searcher@86aee0[collection1] realtime
   [junit4]   2> 492446 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-9223372036854775807&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {deleteByQuery=*:* (-9223372036854775807)} 0 1
   [junit4]   2> 492446 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.u.UpdateHandler start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 492447 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_2,generation=2}
   [junit4]   2> 492447 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.c.SolrCore newest commit generation = 2
   [junit4]   2> 492447 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.s.SolrIndexSearcher Opening Searcher@78d575[collection1] main
   [junit4]   2> 492448 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.u.UpdateHandler end_commit_flush
   [junit4]   2> 492448 INFO  (searcherExecutor-2426-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@78d575[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 492448 INFO  (TEST-TestStressRecovery.testStressRecovery-seed#[301358D17C79C090]) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={} {commit=} 0 1
   [junit4]   2> 492455 INFO  (WRITER2) [    ] o.a.s.u.UpdateHandler start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
   [junit4]   2> 492468 INFO  (WRITER2) [    ] o.a.s.c.SolrCore SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 492468 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=5&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (WRITER2) [    ] o.a.s.u.UpdateHandler end_commit_flush
   [junit4]   2> 492469 INFO  (WRITER1) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-2&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[0 (-2)]} 0 13
   [junit4]   2> 492469 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:10&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:16&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=2&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=13&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=21&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:15&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=11&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=14&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=22&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=16&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=21&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=21&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=7&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=13&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (WRITER0) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-1&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[0 (-1)]} 0 14
   [junit4]   2> 492469 INFO  (WRITER8) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-5&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[9 (-5)]} 0 1
   [junit4]   2> 492469 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:7&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492469 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:15&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=6&wt=json} status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492469 INFO  (WRITER2) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={} {commit=} 0 13
   [junit4]   2> 492469 INFO  (WRITER6) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-3&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[18 (-3)]} 0 13
   [junit4]   2> 492469 INFO  (WRITER7) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-4&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[0 (-4)]} 0 0
   [junit4]   2> 492472 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:0&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:2&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=15&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=13&wt=json} status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:0&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=10&wt=json} status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=2&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:8&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=19&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492471 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=2&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0 
   [junit4]   2> 492470 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:19&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:5&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:13&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:22&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=13&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:16&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=5&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=9&wt=json} status=0 QTime=0 
   [junit4]   2> 492472 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:11&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492472 INFO  (WRITER7) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-6&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[18 (-6)]} 0 0
   [junit4]   2> 492474 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=10&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:2&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=1&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=7&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=10&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=1&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:5&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:20&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492473 INFO  (WRITER2) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={_version_=-7&update.distrib=FROMLEADER&versions=true&wt=json&indent=true} {delete=[18 (-7)]} 0 0
   [junit4]   2> 492473 INFO  (WRITER6) [    ] o.a.s.u.UpdateHandler start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 492473 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:4&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=14&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:9&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=7&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=19&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492475 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:12&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492474 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=11&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:5&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=19&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=16&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=16&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=13&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:10&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=6&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=15&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=9&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=4&wt=json} status=0 QTime=0 
   [junit4]   2> 492478 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492476 INFO  (WRITER6) [    ] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_2,generation=2}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@245d86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@865fad),segFN=segments_3,generation=3}
   [junit4]   2> 492478 INFO  (WRITER6) [    ] o.a.s.c.SolrCore newest commit generation = 3
   [junit4]   2> 492476 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492478 INFO  (WRITER6) [    ] o.a.s.s.SolrIndexSearcher Opening Searcher@3c8d81[collection1] main
   [junit4]   2> 492478 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=10&wt=json} status=0 QTime=0 
   [junit4]   2> 492478 INFO  (WRITER6) [    ] o.a.s.u.UpdateHandler end_commit_flush
   [junit4]   2> 492478 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492478 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492478 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:14&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492479 INFO  (searcherExecutor-2426-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c8d81[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 492477 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=1&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (WRITER6) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 492477 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:7&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=6&wt=json} status=0 QTime=0 
   [junit4]   2> 492477 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=7&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=6&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=22&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:11&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=15&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:13&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=8 
   [junit4]   2> 492488 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=2&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:13&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=15&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=15&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492488 INFO  (WRITER8) [    ] o.a.s.u.UpdateHandler start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 492489 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:19&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492480 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=4&wt=json} status=0 QTime=1 
   [junit4]   2> 492480 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:21&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492479 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:16&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=20&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:15&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=22&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:0&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:19&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:20&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=23&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=10&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (WRITER8) [    ] o.a.s.u.UpdateHandler No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 492488 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:20&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492488 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:20&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=16&wt=json} status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=20&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (WRITER8) [    ] o.a.s.c.SolrCore SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:20&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:15&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:13&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:12&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=17&wt=json} status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:19&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492490 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=20&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=8&wt=json} status=0 QTime=0 
   [junit4]   2> 492489 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=11&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER5) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=2&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=24&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:8&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:13&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=21&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492491 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=3&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=14&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492491 INFO  (WRITER8) [    ] o.a.s.u.UpdateHandler end_commit_flush
   [junit4]   2> 492491 INFO  (READER7) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:14&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492493 INFO  (WRITER8) [    ] o.a.s.u.p.LogUpdateProcessor [collection1] webapp=null path=null params={} {commit=} 0 5
   [junit4]   2> 492493 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER4) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:4&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER1) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=0&wt=json} status=0 QTime=0 
   [junit4]   2> 492493 INFO  (READER3) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:14&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER2) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=9&wt=json} status=0 QTime=0 
   [junit4]   2> 492492 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:0&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={q=id:18&omitHeader=true&wt=json} hits=0 status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=22&wt=json} status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=11&wt=json} status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER0) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=18&wt=json} status=0 QTime=0 
   [junit4]   2> 492494 INFO  (READER6) [    ] o.a.s.c.S.Request [collection1] webapp=null path=null params={qt=/get&ids=4&wt=json} status=0 QTime=0 
   [junit4] 

[...truncated too long message...]

7)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:193)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 683434 WARN  (zkCallback-258-thread-6-processing-n:127.0.0.1:60839_) [n:127.0.0.1:60839_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 683434 ERROR (TEST-CdcrReplicationHandlerTest.doTest-seed#[301358D17C79C090]-EventThread) [n:127.0.0.1:60839_    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@1662bba rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1b95a76[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 32]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:193)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 683435 INFO  (TEST-CdcrReplicationHandlerTest.doTest-seed#[301358D17C79C090]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@58a640{SSL-http/1.1}{127.0.0.1:60839}
   [junit4]   2> 683435 INFO  (TEST-CdcrReplicationHandlerTest.doTest-seed#[301358D17C79C090]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@9aa666{/,null,UNAVAILABLE}
   [junit4]   2> 683437 INFO  (TEST-CdcrReplicationHandlerTest.doTest-seed#[301358D17C79C090]) [n:127.0.0.1:60839_ c:source_collection s:shard1 r:core_node2 x:source_collection_shard1_replica2] o.a.s.SolrTestCaseJ4 ###Ending doTest
   [junit4]   2> 683438 INFO  (TEST-CdcrReplicationHandlerTest.doTest-seed#[301358D17C79C090]) [n:127.0.0.1:60839_ c:source_collection s:shard1 r:core_node2 x:source_collection_shard1_replica2] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44142 44142
   [junit4]   2> 683445 INFO  (Thread-682) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44142 44142
   [junit4]   2> 683446 WARN  (Thread-682) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	29	/solr/aliases.json
   [junit4]   2> 	29	/solr/clusterstate.json
   [junit4]   2> 	8	/solr/configs/conf1
   [junit4]   2> 	6	/solr/collections/source_collection/state.json
   [junit4]   2> 	5	/solr/collections/source_collection/leaders/shard1
   [junit4]   2> 	2	/solr/collections/tmp_collection/leaders/shard1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/collections/source_collection/cdcr/state/buffer
   [junit4]   2> 	7	/solr/collections/source_collection/cdcr/state/process
   [junit4]   2> 	2	/solr/collections/source_collection/leader_elect/shard1/election/94272467685933088-core_node2-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/94272467685933061-127.0.0.1:56963_-n_0000000000
   [junit4]   2> 	2	/solr/collections/tmp_collection/cdcr/state/buffer
   [junit4]   2> 	2	/solr/collections/source_collection/leader_elect/shard1/election/94272467685933061-core_node2-n_0000000000
   [junit4]   2> 	2	/solr/collections/tmp_collection/cdcr/state/process
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	28	/solr/live_nodes
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CdcrReplicationHandlerTest -Dtests.method=doTest -Dtests.seed=301358D17C79C090 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_IE -Dtests.timezone=Pacific/Bougainville -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  387s J1 | CdcrReplicationHandlerTest.doTest <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([301358D17C79C090:9757E07511C2D329]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
   [junit4]    > 	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForRecoveriesToFinish(BaseCdcrDistributedZkTest.java:465)
   [junit4]    > 	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.clearSourceCollection(BaseCdcrDistributedZkTest.java:319)
   [junit4]    > 	at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTestPartialReplication(CdcrReplicationHandlerTest.java:86)
   [junit4]    > 	at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest(CdcrReplicationHandlerTest.java:51)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:966)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:941)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 683454 INFO  (SUITE-CdcrReplicationHandlerTest-seed#[301358D17C79C090]-worker) [n:127.0.0.1:60839_ c:source_collection s:shard1 r:core_node2 x:source_collection_shard1_replica2] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CdcrReplicationHandlerTest_301358D17C79C090-001
   [junit4]   2> Aug 02, 2015 2:54:45 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=21678, maxDocsPerChunk=9, blockSize=123), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=21678, blockSize=123)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_IE, timezone=Pacific/Bougainville
   [junit4]   2> NOTE: Linux 3.16.0-44-generic i386/Oracle Corporation 1.8.0_51 (32-bit)/cpus=12,threads=1,free=129227648,total=250789888
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsWriteToMultipleCollectionsTest, CursorMarkTest, TestHighlightDedupGrouping, ZkStateWriterTest, FieldAnalysisRequestHandlerTest, ChaosMonkeySafeLeaderTest, RemoteQueryErrorTest, TestScoreJoinQPScore, TestSearchPerf, TestJsonFacets, TestFastWriter, FieldMutatingUpdateProcessorTest, TestPivotHelperCode, OutOfBoxZkACLAndCredentialsProvidersTest, TestManagedSchema, TestRestManager, TestIntervalFaceting, BitVectorTest, SolrPluginUtilsTest, TestLFUCache, RecoveryAfterSoftCommitTest, NoCacheHeaderTest, TestClassNameShortening, BlockDirectoryTest, TestInitQParser, AnalyticsMergeStrategyTest, HdfsUnloadDistributedZkTest, TestCloudPivotFacet, DirectSolrConnectionTest, FileUtilsTest, TestReplicationHandlerBackup, SliceStateTest, TestHashPartitioner, TestStressVersions, TestExpandComponent, CacheHeaderTest, BadComponentTest, TestNRTOpen, DeleteLastCustomShardedReplicaTest, TestSolrCoreProperties, CdcrReplicationHandlerTest]
   [junit4] Completed [179/529] on J1 in 387.23s, 1 test, 1 failure <<< FAILURES!

[...truncated 1106 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:716: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:660: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:59: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:230: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:514: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1429: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:984: There were test failures: 529 suites, 2092 tests, 9 suite-level errors, 1 error, 1 failure, 65 ignored (30 assumptions)

Total time: 53 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_51) - Build # 13696 - Failure!

Posted by Yonik Seeley <ys...@gmail.com>.
Good catch, thanks!
-Yonik

On Sat, Aug 1, 2015 at 10:26 PM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> I committed a fix to the CdcrUpdateLogTest to reset the test hooks.
>
> On Sun, Aug 2, 2015 at 10:42 AM, Shalin Shekhar Mangar
> <sh...@gmail.com> wrote:
>> Yeah, looks like the Cdcr tests is enabling the test hooks in a static
>> context and not cleaning up? The TestStressRecovery ends up calling
>> the tests hook at CdcrUpdateLogTest.java line 444. Look at the failure
>> stack trace:
>>
>>    [junit4]   2> NOTE: reproduce with: ant test
>> -Dtestcase=TestStressRecovery -Dtests.method=testStressRecovery
>> -Dtests.seed=301358D17C79C090 -Dtests.multiplier=3 -Dtests.slow=true
>> -Dtests.locale=lt -Dtests.timezone=America/Havana -Dtests.asserts=true
>> -Dtests.file.encoding=UTF-8
>>    [junit4] ERROR   92.3s J2 | TestStressRecovery.testStressRecovery <<<
>>    [junit4]    > Throwable #1:
>> java.util.concurrent.ExecutionException: java.lang.AssertionError
>>    [junit4]    > at
>> __randomizedtesting.SeedInfo.seed([301358D17C79C090:8A29318CE3917F9E]:0)
>>    [junit4]    > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>    [junit4]    > at java.util.concurrent.FutureTask.get(FutureTask.java:206)
>>    [junit4]    > at
>> org.apache.solr.search.TestStressRecovery.testStressRecovery(TestStressRecovery.java:370)
>>    [junit4]    > at java.lang.Thread.run(Thread.java:745)
>>    [junit4]    > Caused by: java.lang.AssertionError
>>    [junit4]    > at
>> org.apache.solr.update.CdcrUpdateLogTest$1.run(CdcrUpdateLogTest.java:444)
>>    [junit4]    > at
>> org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1320)
>>    [junit4]    > at
>> org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1258)
>>    [junit4]    > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>    [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>    [junit4]    > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>
>> On Sun, Aug 2, 2015 at 10:22 AM, Yonik Seeley <ys...@gmail.com> wrote:
>>> It's been two years since this test has failed, and it seems to have
>>> failed multiple times recently.  The test itself hasn't changed.
>>> Anyone have any ideas what has changed here?
>>>
>>> Hmmm, looking at the output, perhaps this is this just bleed-over from
>>> other failing tests?
>>>
>>> -Yonik
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_51) - Build # 13696 - Failure!

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I committed a fix to the CdcrUpdateLogTest to reset the test hooks.

On Sun, Aug 2, 2015 at 10:42 AM, Shalin Shekhar Mangar
<sh...@gmail.com> wrote:
> Yeah, looks like the Cdcr tests is enabling the test hooks in a static
> context and not cleaning up? The TestStressRecovery ends up calling
> the tests hook at CdcrUpdateLogTest.java line 444. Look at the failure
> stack trace:
>
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtestcase=TestStressRecovery -Dtests.method=testStressRecovery
> -Dtests.seed=301358D17C79C090 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=lt -Dtests.timezone=America/Havana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   92.3s J2 | TestStressRecovery.testStressRecovery <<<
>    [junit4]    > Throwable #1:
> java.util.concurrent.ExecutionException: java.lang.AssertionError
>    [junit4]    > at
> __randomizedtesting.SeedInfo.seed([301358D17C79C090:8A29318CE3917F9E]:0)
>    [junit4]    > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>    [junit4]    > at java.util.concurrent.FutureTask.get(FutureTask.java:206)
>    [junit4]    > at
> org.apache.solr.search.TestStressRecovery.testStressRecovery(TestStressRecovery.java:370)
>    [junit4]    > at java.lang.Thread.run(Thread.java:745)
>    [junit4]    > Caused by: java.lang.AssertionError
>    [junit4]    > at
> org.apache.solr.update.CdcrUpdateLogTest$1.run(CdcrUpdateLogTest.java:444)
>    [junit4]    > at
> org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1320)
>    [junit4]    > at
> org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1258)
>    [junit4]    > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>    [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>    [junit4]    > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> On Sun, Aug 2, 2015 at 10:22 AM, Yonik Seeley <ys...@gmail.com> wrote:
>> It's been two years since this test has failed, and it seems to have
>> failed multiple times recently.  The test itself hasn't changed.
>> Anyone have any ideas what has changed here?
>>
>> Hmmm, looking at the output, perhaps this is this just bleed-over from
>> other failing tests?
>>
>> -Yonik
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.



-- 
Regards,
Shalin Shekhar Mangar.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_51) - Build # 13696 - Failure!

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
Yeah, looks like the Cdcr tests is enabling the test hooks in a static
context and not cleaning up? The TestStressRecovery ends up calling
the tests hook at CdcrUpdateLogTest.java line 444. Look at the failure
stack trace:

   [junit4]   2> NOTE: reproduce with: ant test
-Dtestcase=TestStressRecovery -Dtests.method=testStressRecovery
-Dtests.seed=301358D17C79C090 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=lt -Dtests.timezone=America/Havana -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   92.3s J2 | TestStressRecovery.testStressRecovery <<<
   [junit4]    > Throwable #1:
java.util.concurrent.ExecutionException: java.lang.AssertionError
   [junit4]    > at
__randomizedtesting.SeedInfo.seed([301358D17C79C090:8A29318CE3917F9E]:0)
   [junit4]    > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
   [junit4]    > at java.util.concurrent.FutureTask.get(FutureTask.java:206)
   [junit4]    > at
org.apache.solr.search.TestStressRecovery.testStressRecovery(TestStressRecovery.java:370)
   [junit4]    > at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > at
org.apache.solr.update.CdcrUpdateLogTest$1.run(CdcrUpdateLogTest.java:444)
   [junit4]    > at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1320)
   [junit4]    > at
org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1258)
   [junit4]    > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]    > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

On Sun, Aug 2, 2015 at 10:22 AM, Yonik Seeley <ys...@gmail.com> wrote:
> It's been two years since this test has failed, and it seems to have
> failed multiple times recently.  The test itself hasn't changed.
> Anyone have any ideas what has changed here?
>
> Hmmm, looking at the output, perhaps this is this just bleed-over from
> other failing tests?
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
Regards,
Shalin Shekhar Mangar.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_51) - Build # 13696 - Failure!

Posted by Yonik Seeley <ys...@gmail.com>.
It's been two years since this test has failed, and it seems to have
failed multiple times recently.  The test itself hasn't changed.
Anyone have any ideas what has changed here?

Hmmm, looking at the output, perhaps this is this just bleed-over from
other failing tests?

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org