[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1033 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1033/

3 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=22727, name=zkCallback-553-thread-15, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=22727, name=zkCallback-553-thread-15, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
116 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=22487, name=searcherExecutor-3572-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=19958, name=qtp129174011-19958-selector-ServerConnectorManager@9c681d/1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=20335, name=searcherExecutor-1760-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=19965, name=Scheduler-1630664890, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=19981, name=searcherExecutor-1498-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=20257, name=searcherExecutor-1666-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=22474, name=searcherExecutor-3577-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=22980, name=zkCallback-606-thread-12-processing-n:127.0.0.1:60958_, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1028)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1014)         at org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1597)         at org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:443)         at org.apache.solr.cloud.ZkController.access$100(ZkController.java:89)         at org.apache.solr.cloud.ZkController$1.command(ZkController.java:272)         at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)         at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)         at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)         at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)         at java.util.concurrent.FutureTask.run(FutureTask.java:262)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=22440, name=searcherExecutor-3525-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=22301, name=qtp129174011-22301, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=22229, name=searcherExecutor-3399-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=20435, name=qtp2011327979-20435-selector-ServerConnectorManager@1be9f792/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=22563, name=searcherExecutor-3606-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=22228, name=searcherExecutor-3404-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=20666, name=searcherExecutor-1988-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=20437, name=qtp2011327979-20437-selector-ServerConnectorManager@1be9f792/1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=22465, name=searcherExecutor-3503-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=20469, name=Thread-17402, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1238)   19) Thread[id=20534, name=searcherExecutor-1902-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=21184, name=qtp1015471035-21184, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1014)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:970)         at org.apache.solr.cloud.ZkController.register(ZkController.java:909)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)         at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)         at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:696)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:820)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:750)         at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:617)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:212)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:221)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:180)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)         at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)         at org.eclipse.jetty.server.Server.handle(Server.java:499)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=20024, name=Scheduler-141426940, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=20285, name=searcherExecutor-1718-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=22438, name=searcherExecutor-3517-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=20016, name=qtp1015471035-20016-selector-ServerConnectorManager@26c9cef5/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=22386, name=searcherExecutor-3445-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=22066, name=qtp1015471035-22066, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=20645, name=searcherExecutor-1978-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=20151, name=searcherExecutor-1600-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=22481, name=searcherExecutor-3568-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=20139, name=searcherExecutor-1587-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=20442, name=Scheduler-674380372, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=22215, name=searcherExecutor-3393-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=20036, name=Thread-17187, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1238)   34) Thread[id=20350, name=searcherExecutor-1772-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=22427, name=searcherExecutor-3489-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=20543, name=searcherExecutor-1907-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=22290, name=qtp1015471035-22290, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1041)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1014)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:970)         at org.apache.solr.cloud.ZkController.register(ZkController.java:909)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)         at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)         at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:696)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:820)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:750)         at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:617)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:212)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:221)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:180)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)         at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)         at org.eclipse.jetty.server.Server.handle(Server.java:499)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=22394, name=searcherExecutor-3470-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=20053, name=Scheduler-969142536, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=22231, name=searcherExecutor-3406-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=22470, name=searcherExecutor-3578-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=22326, name=qtp1015471035-22326, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=22323, name=qtp129174011-22323, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=23081, name=zkCallback-606-thread-12-processing-n:127.0.0.1:60958_-SendThread(127.0.0.1:50840), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   45) Thread[id=20241, name=searcherExecutor-1668-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=20509, name=searcherExecutor-1879-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=20487, name=Scheduler-849255626, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=20671, name=searcherExecutor-1998-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   49) Thread[id=20025, name=org.eclipse.jetty.server.session.HashSessionManager@5821be24Timer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   50) Thread[id=22505, name=searcherExecutor-3573-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   51) Thread[id=20171, name=searcherExecutor-1636-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   52) Thread[id=20040, name=searcherExecutor-1520-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   53) Thread[id=20149, name=searcherExecutor-1597-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   54) Thread[id=20263, name=searcherExecutor-1694-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   55) Thread[id=20045, name=Scheduler-2069119407, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   56) Thread[id=22264, name=qtp129174011-22264, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   57) Thread[id=20544, name=searcherExecutor-1897-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   58) Thread[id=22496, name=searcherExecutor-3562-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   59) Thread[id=20152, name=searcherExecutor-1595-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   60) Thread[id=22391, name=searcherExecutor-3465-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   61) Thread[id=20061, name=searcherExecutor-1532-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   62) Thread[id=22430, name=searcherExecutor-3483-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   63) Thread[id=20249, name=searcherExecutor-1669-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   64) Thread[id=21237, name=qtp2011327979-21237, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1028)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1014)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:970)         at org.apache.solr.cloud.ZkController.register(ZkController.java:909)         at org.apache.solr.cloud.ZkController.register(ZkController.java:852)         at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)         at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)         at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:696)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:820)         at org.apache.solr.core.CoreContainer.create(CoreContainer.java:750)         at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:617)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:212)         at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:192)         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:221)         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:180)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)         at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)         at org.eclipse.jetty.server.Server.handle(Server.java:499)         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   65) Thread[id=23051, name=zkCallback-606-thread-12-processing-n:127.0.0.1:60958_-EventThread, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)         at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:377)         at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:250)         at org.apache.solr.cloud.ZkController.publish(ZkController.java:1228)         at org.apache.solr.cloud.ZkController.publish(ZkController.java:1140)         at org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:393)         at org.apache.solr.cloud.ZkController.access$100(ZkController.java:89)         at org.apache.solr.cloud.ZkController$1.command(ZkController.java:272)         at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:158)         at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:56)         at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:132)         at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)   66) Thread[id=20547, name=searcherExecutor-1908-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   67) Thread[id=20481, name=searcherExecutor-1866-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   68) Thread[id=20438, name=qtp2011327979-20438-acceptor-0@61111b68-ServerConnector@1d7ca08f{SSL-http/1.1}{127.0.0.1:60958}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   69) Thread[id=22569, name=searcherExecutor-3623-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   70) Thread[id=20531, name=searcherExecutor-1896-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   71) Thread[id=20258, name=searcherExecutor-1689-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   72) Thread[id=20166, name=searcherExecutor-1630-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   73) Thread[id=22371, name=searcherExecutor-3433-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   74) Thread[id=20662, name=searcherExecutor-1983-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   75) Thread[id=20506, name=searcherExecutor-1881-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   76) Thread[id=22222, name=searcherExecutor-3395-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   77) Thread[id=20499, name=searcherExecutor-1876-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   78) Thread[id=19957, name=qtp129174011-19957-selector-ServerConnectorManager@9c681d/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   79) Thread[id=23082, name=zkCallback-606-thread-12-processing-n:127.0.0.1:60958_-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   80) Thread[id=22538, name=searcherExecutor-3604-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   81) Thread[id=20363, name=searcherExecutor-1791-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   82) Thread[id=20225, name=searcherExecutor-1660-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   83) Thread[id=20533, name=searcherExecutor-1906-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   84) Thread[id=20668, name=searcherExecutor-1989-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   85) Thread[id=20399, name=searcherExecutor-1828-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   86) Thread[id=20491, name=searcherExecutor-1871-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   87) Thread[id=22385, name=searcherExecutor-3446-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   88) Thread[id=20340, name=searcherExecutor-1761-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   89) Thread[id=22320, name=qtp129174011-22320, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   90) Thread[id=22434, name=searcherExecutor-3506-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   91) Thread[id=20018, name=qtp1015471035-20018-acceptor-0@12c77bc6-ServerConnector@b324ab8{SSL-http/1.1}{127.0.0.1:51129}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   92) Thread[id=22543, name=searcherExecutor-3622-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   93) Thread[id=22541, name=searcherExecutor-3614-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   94) Thread[id=19959, name=qtp129174011-19959-acceptor-0@5b272816-ServerConnector@1c77483b{SSL-http/1.1}{127.0.0.1:45593}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   95) Thread[id=22418, name=searcherExecutor-3493-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   96) Thread[id=22469, name=searcherExecutor-3556-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   97) Thread[id=22200, name=qtp2011327979-22200, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   98) Thread[id=20505, name=searcherExecutor-1877-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   99) Thread[id=22080, name=qtp129174011-22080, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   100) Thread[id=22373, name=searcherExecutor-3434-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   101) Thread[id=19966, name=org.eclipse.jetty.server.session.HashSessionManager@

[...truncated too long message...]

lectorImpl.select(SelectorImpl.java:102)
   [junit4]    >         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
   [junit4]    >         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
   [junit4]    >         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   90) Thread[id=22427, name=searcherExecutor-3489-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   91) Thread[id=19966, name=org.eclipse.jetty.server.session.HashSessionManager@75f1a51eTimer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   92) Thread[id=20393, name=searcherExecutor-1825-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   93) Thread[id=20543, name=searcherExecutor-1907-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   94) Thread[id=20642, name=searcherExecutor-1973-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   95) Thread[id=20053, name=Scheduler-969142536, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   96) Thread[id=22394, name=searcherExecutor-3470-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B01AFA1951047BBB]:0)
   [junit4] Completed [463/552] on J1 in 3276.63s, 1 test, 3 errors <<< FAILURES!

[...truncated 261 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J1-20151201_084124_734.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/heapdumps/java_pid30103.hprof ...
   [junit4] Heap dump file created [611582236 bytes in 10.024 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J1-20151201_084124_734.syserr
   [junit4] >>> JVM J1: stderr (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "zkCallback-553-thread-13"
   [junit4] <<< JVM J1: EOF ----

[...truncated 17 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:792: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:729: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 552 suites (7 ignored), 2183 tests, 2 suite-level errors, 1 error, 303 ignored (18 assumptions) [seed: B01AFA1951047BBB]

Total time: 213 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1045 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1045/

10 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1721, name=Thread-1644, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1721, name=Thread-1644, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1719, name=Thread-1642, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1719, name=Thread-1642, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	... 2 more


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1722, name=Thread-1645, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1722, name=Thread-1645, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1723, name=Thread-1646, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1723, name=Thread-1646, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1717, name=Thread-1640, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1717, name=Thread-1640, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	... 2 more


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1720, name=Thread-1643, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1720, name=Thread-1643, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	... 2 more


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1718, name=Thread-1641, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1718, name=Thread-1641, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	... 2 more


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
Captured an uncaught exception in thread: Thread[id=1714, name=Thread-1637, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1714, name=Thread-1637, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
	at java.nio.file.Files.newOutputStream(Files.java:172)
	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)




Build Log:
[...truncated 1678 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterThreadsToSegments
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1642,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	... 2 more
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1644,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1640,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	... 2 more
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1643,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	... 2 more
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1641,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	... 2 more
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1637,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1645,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 2:10:10 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-1646,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]   2> 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]   2> 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]   2> 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]   2> 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)
   [junit4]   2> 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]   2> 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]   2> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]   2> 
   [junit4]   2> dec 12, 2015 4:08:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterThreadsToSegments
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
   [junit4]   2>    2) Thread[id=1716, name=Thread-1639, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:227)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2>    3) Thread[id=1698, name=SUITE-TestIndexWriterThreadsToSegments-seed#[F72A04E52535E596], state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1589)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:688)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:685)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:685)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:701)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
   [junit4]   2>    4) Thread[id=8, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2>    5) Thread[id=1715, name=Thread-1638, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:227)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2>    6) Thread[id=1699, name=TEST-TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom-seed#[F72A04E52535E596], state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom(TestIndexWriterThreadsToSegments.java:226)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterThreadsToSegments -Dtests.method=testSegmentCountOnFlushRandom -Dtests.seed=F72A04E52535E596 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   7199s J1 | TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]   2> dec 12, 2015 4:08:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> dec 12, 2015 4:09:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 3 threads leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterThreadsToSegments: 
   [junit4]   2>    1) Thread[id=1715, name=Thread-1638, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:227)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2>    2) Thread[id=1699, name=TEST-TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom-seed#[F72A04E52535E596], state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom(TestIndexWriterThreadsToSegments.java:226)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    3) Thread[id=1716, name=Thread-1639, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:227)
   [junit4]   2>         at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:355)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> dec 12, 2015 4:09:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=1715, name=Thread-1638, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>    2) Thread[id=1699, name=TEST-TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom-seed#[F72A04E52535E596], state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2>    3) Thread[id=1716, name=Thread-1639, state=WAITING, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001
   [junit4]   2> Dec 13, 2015 12:09:02 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {field=Lucene50(blocksize=128)}, docValues:{}, sim=DefaultSimilarity, locale=sr_RS_#Latn, timezone=America/Santa_Isabel
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=130871024,total=402128896
   [junit4]   2> NOTE: All tests run in this JVM: [TestTermScorer, TestDocCount, TestSortedSetSortField, TestIndexWriterLockRelease, TestIndexWriterExceptions, TestBoostQuery, TestDisjunctionMaxQuery, TestIndexWriterDeleteByQuery, Test2BNumericDocValues, TestSegmentMerger, TestDemoParallelLeafReader, TestFlex, TestFastDecompressionMode, TestBoolean2, TestControlledRealTimeReopenThread, TestTermVectorsReader, TestRecyclingIntBlockAllocator, TestBlendedTermQuery, TestFSTs, TestFieldValueFilter, TestIndexWriterThreadsToSegments]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterThreadsToSegments -Dtests.seed=F72A04E52535E596 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr_RS_#Latn -Dtests.timezone=America/Santa_Isabel -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestIndexWriterThreadsToSegments (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1721, name=Thread-1644, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ks.fdx: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1719, name=Thread-1642, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	... 2 moreThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1722, name=Thread-1645, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kp.fdx: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1723, name=Thread-1646, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1kt.fdt: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:111)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thr

[...truncated too long message...]

> 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	... 2 moreThrowable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1718, name=Thread-1641, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	... 2 moreThrowable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1714, name=Thread-1637, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F72A04E52535E596]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.nio.file.FileSystemException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterThreadsToSegments_F72A04E52535E596-001/tempDir-001/_1ku.fdx: Too many open files
   [junit4]    > 	at org.apache.lucene.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:49)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
   [junit4]    > 	at org.apache.lucene.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:161)
   [junit4]    > 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
   [junit4]    > 	at java.nio.file.Files.newOutputStream(Files.java:172)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:276)
   [junit4]    > 	at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:225)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:593)
   [junit4]    > 	at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
   [junit4]    > 	at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
   [junit4]    > 	at org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
   [junit4]    > 	at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat.fieldsWriter(AssertingStoredFieldsFormat.java:49)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:81)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:258)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:295)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:207)
   [junit4] Completed [366/414 (1!)] on J1 in 7220.20s, 3 tests, 10 errors <<< FAILURES!

[...truncated 170 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1010: There were test failures: 414 suites (1 ignored), 3345 tests, 9 suite-level errors, 1 error, 29 ignored (25 assumptions) [seed: F72A04E52535E596]

Total time: 157 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1044 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1044/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:41741/_v/vj: Could not load collection from ZK:halfcollectionblocker

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41741/_v/vj: Could not load collection from ZK:halfcollectionblocker
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:295)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:412)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:162)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
Unable to restart (#4): CloudJettyRunner [url=http://127.0.0.1:56785/be/c/collection1]

Stack Trace:
java.lang.AssertionError: Unable to restart (#4): CloudJettyRunner [url=http://127.0.0.1:56785/be/c/collection1]
	at __randomizedtesting.SeedInfo.seed([91E270C9DE67D054:19B64F13709BBDAC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:104)
	at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10372 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/init-core-data-001
   [junit4]   2> 169612 INFO  (SUITE-RollingRestartTest-seed#[91E270C9DE67D054]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 169613 INFO  (SUITE-RollingRestartTest-seed#[91E270C9DE67D054]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /be/c
   [junit4]   2> 169627 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 169628 INFO  (Thread-3945) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 169628 INFO  (Thread-3945) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 169728 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkTestServer start zk server on port:55353
   [junit4]   2> 169729 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 169758 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 169762 INFO  (zkCallback-83-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@16b7755f name:ZooKeeperConnection Watcher:127.0.0.1:55353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 169762 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 169762 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 169762 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 169766 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 169767 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 169773 INFO  (zkCallback-84-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2917ca48 name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 169773 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 169774 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 169774 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 169777 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 169779 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 169781 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 169783 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 169783 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 169788 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 169788 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 169791 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 169791 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 169794 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 169794 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 169799 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 169799 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 169802 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 169803 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 169805 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 169805 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 169807 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 169807 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 169810 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 169810 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 169812 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 169812 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 169814 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 169814 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 170178 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1
   [junit4]   2> 170180 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 170183 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5ce1ccdd{/be/c,null,AVAILABLE}
   [junit4]   2> 170184 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6c50f0ce{HTTP/1.1}{127.0.0.1:39963}
   [junit4]   2> 170184 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.Server Started @173124ms
   [junit4]   2> 170184 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/be/c, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/tempDir-001/control/data, hostPort=39963}
   [junit4]   2> 170185 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4dfd245f
   [junit4]   2> 170185 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001'
   [junit4]   2> 170185 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 170185 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 170186 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 170186 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 170188 INFO  (zkCallback-85-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@66490ef3 name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 170188 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 170188 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 170190 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 170190 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/solr.xml
   [junit4]   2> 170204 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores
   [junit4]   2> 170205 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer New CoreContainer 1784043863
   [junit4]   2> 170205 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001]
   [junit4]   2> 170205 WARN  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/lib
   [junit4]   2> 170205 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 170206 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 170207 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 170207 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 170207 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55353/solr
   [junit4]   2> 170207 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 170207 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 170208 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 170210 INFO  (zkCallback-87-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1323271c name:ZooKeeperConnection Watcher:127.0.0.1:55353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 170210 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 170211 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 170214 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 170216 INFO  (zkCallback-88-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6e13f60e name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 170216 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 170218 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 170220 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 170224 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 170227 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 170229 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 170232 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 170235 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 170237 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 170239 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 170241 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39963_be%2Fc
   [junit4]   2> 170242 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:39963_be%2Fc
   [junit4]   2> 170244 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 170246 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 170248 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 170249 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000
   [junit4]   2> 170250 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39963_be%2Fc
   [junit4]   2> 170250 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 170252 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer Overseer (id=95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) starting
   [junit4]   2> 170254 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 170275 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 170275 INFO  (OverseerCollectionConfigSetProcessor-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 170276 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 170278 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 170314 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 170315 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 170316 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores
   [junit4]   2> 170317 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {coreNodeName=, loadOnStartup=true, transient=false, configSetProperties=configsetprops.json, schema=schema.xml, dataDir=data/, collection=control_collection, name=collection1, shard=, config=solrconfig.xml}
   [junit4]   2> 170317 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1
   [junit4]   2> 170318 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 170319 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 170319 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 170320 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 170320 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 170324 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:39963_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:39963/be/c"} current state version: 0
   [junit4]   2> 170324 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "node_name":"127.0.0.1:39963_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "base_url":"http://127.0.0.1:39963/be/c"}
   [junit4]   2> 170325 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 170325 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 170325 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 170328 INFO  (zkCallback-88-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 171325 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 171326 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 171327 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 171327 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 171328 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 171328 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1'
   [junit4]   2> 171328 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 171328 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 171331 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 171339 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 171354 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 171384 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 171394 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 171670 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 171679 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 171682 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 171705 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 171710 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 171713 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 171714 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 171714 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 171714 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 171715 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 171715 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 171716 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 171717 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 171717 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore solr.StandardDirectoryFactory
   [junit4]   2> 171724 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/]
   [junit4]   2> 171724 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@531fb850
   [junit4]   2> 171725 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data
   [junit4]   2> 171726 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/index/
   [junit4]   2> 171726 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 171726 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/index
   [junit4]   2> 171727 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=93.4091796875, floorSegmentMB=1.140625, forceMergeDeletesPctAllowed=19.040737700900713, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.798895882218802
   [junit4]   2> 171735 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 171735 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 171736 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/
   [junit4]   2> 171742 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 171742 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 171743 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 171744 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 171744 WARN  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 171748 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 171749 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 171750 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 171752 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 171757 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/plugins,/admin/luke,/admin/mbeans,/dump,/admin/segments,/update/csv,standard,/schema,/replication,/config,/admin/properties,/update,/admin/system,/get,/update/json/docs,/admin/logging,/admin/ping,/update/json,/admin/threads,/admin/file
   [junit4]   2> 171759 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 171760 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 171760 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 171760 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 171760 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 171761 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=73.013671875, floorSegmentMB=1.62890625, forceMergeDeletesPctAllowed=22.559599334711724, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 171763 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 171763 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 171763 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@1a72b5ed[collection1] main
   [junit4]   2> 171763 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 171764 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 171764 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 171765 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 171766 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 171767 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 171767 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 171767 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 171767 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1520326656622854144
   [junit4]   2> 171769 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1520326656622854144
   [junit4]   2> 171769 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 171770 INFO  (searcherExecutor-285-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1a72b5ed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 171770 INFO  (coreLoadExecutor-284-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 171771 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:39963/be/c collection:control_collection shard:shard1
   [junit4]   2> 171773 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 171778 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95020415899533316-core_node1-n_0000000000
   [junit4]   2> 171779 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 171781 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 171782 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 171782 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:39963/be/c/collection1/
   [junit4]   2> 171782 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 171782 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:39963/be/c/collection1/ has no replicas
   [junit4]   2> 171782 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 171783 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 171788 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 171791 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:39963/be/c/collection1/ shard1
   [junit4]   2> 171792 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:39963/be/c",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 171821 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 171822 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 171824 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@35fda302 name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 171824 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 171824 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 171824 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 171829 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 171896 INFO  (zkCallback-88-thread-1-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 171896 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 171943 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:39963/be/c/collection1/ and leader is http://127.0.0.1:39963/be/c/collection1/
   [junit4]   2> 171943 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39963/be/c
   [junit4]   2> 171943 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 171943 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 171943 INFO  (coreZkRegister-278-thread-1-processing-n:127.0.0.1:39963_be%2Fc s:shard1 c:control_collection x:collection1 r:core_node1) [n:127.0.0.1:39963_be%2Fc c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 171946 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:39963_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"http://127.0.0.1:39963/be/c"} current state version: 2
   [junit4]   2> 171947 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "node_name":"127.0.0.1:39963_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"http://127.0.0.1:39963/be/c"}
   [junit4]   2> 172051 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 172051 INFO  (zkCallback-88-thread-2-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 172190 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores/collection1
   [junit4]   2> 172191 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001
   [junit4]   2> 172191 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 172194 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3951e1e6{/be/c,null,AVAILABLE}
   [junit4]   2> 172194 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4111f10d{HTTP/1.1}{127.0.0.1:57469}
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.e.j.s.Server Started @175134ms
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=57469, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores, hostContext=/be/c, solrconfig=solrconfig.xml, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/tempDir-001/jetty1}
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4dfd245f
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001'
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 172195 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 172196 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 172196 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 172198 INFO  (zkCallback-91-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@32c8d2ed name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 172198 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 172199 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 172200 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 172200 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/solr.xml
   [junit4]   2> 172217 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores
   [junit4]   2> 172218 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer New CoreContainer 1400400768
   [junit4]   2> 172218 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001]
   [junit4]   2> 172218 WARN  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/lib
   [junit4]   2> 172219 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 172220 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 172221 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 172221 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 172221 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55353/solr
   [junit4]   2> 172221 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 172221 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 172222 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 172223 INFO  (zkCallback-93-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@73c68f60 name:ZooKeeperConnection Watcher:127.0.0.1:55353 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 172224 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 172224 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 172227 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 172230 INFO  (zkCallback-94-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7212f0ad name:ZooKeeperConnection Watcher:127.0.0.1:55353/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 172230 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 172238 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 173247 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57469_be%2Fc
   [junit4]   2> 173247 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:57469_be%2Fc
   [junit4]   2> 173249 INFO  (zkCallback-94-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 173250 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 173250 INFO  (zkCallback-88-thread-2-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 173251 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 173252 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95020415899533320-127.0.0.1:57469_be%2Fc-n_0000000001
   [junit4]   2> 173254 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000 to know if I could be the leader
   [junit4]   2> 173275 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 173275 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 173276 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores
   [junit4]   2> 173277 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {coreNodeName=, loadOnStartup=true, config=solrconfig.xml, transient=false, name=collection1, shard=, collection=collection1, dataDir=data/, schema=schema.xml, configSetProperties=configsetprops.json}
   [junit4]   2> 173277 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores/collection1
   [junit4]   2> 173278 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 173280 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 173280 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 173280 INFO  (TEST-RollingRestartTest.test-seed#[91E270C9DE67D054]) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 173280 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 173283 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 173284 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "node_name":"127.0.0.1:57469_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"http://127.0.0.1:57469/be/c"} current state version: 3
   [junit4]   2> 173284 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "node_name":"127.0.0.1:57469_be%2Fc",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null,
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "base_url":"http://127.0.0.1:57469/be/c"}
   [junit4]   2> 173284 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 173284 INFO  (OverseerStateUpdate-95020415899533316-127.0.0.1:39963_be%2Fc-n_0000000000) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 173389 INFO  (zkCallback-88-thread-2-processing-n:127.0.0.1:39963_be%2Fc) [n:127.0.0.1:39963_be%2Fc    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 173389 INFO  (zkCallback-94-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 173389 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 174284 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 174285 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 174285 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 174285 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/collection1
   [junit4]   2> 174286 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 174286 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.RollingRestartTest_91E270C9DE67D054-001/shard-1-001/cores/collection1'
   [junit4]   2> 174287 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 174287 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 174290 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 174298 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 174315 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 174362 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 174371 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 174633 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 174643 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 174646 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 174663 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 174668 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 174671 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 174672 WARN  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 174672 WARN  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:57469_be%2Fc) [n:127.0.0.1:57469_be%2Fc c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 174673 INFO  (coreLoadExecutor-295-thread-1-processing-n:127.0.0.1:5

[...truncated too long message...]

p://127.0.0.1:48329/_v/vj, http://127.0.0.1:44086/_v/vj, http://127.0.0.1:35231/_v/vj, http://127.0.0.1:46864/_v/vj]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([91E270C9DE67D054]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:891)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:41741/_v/vj, http://127.0.0.1:48329/_v/vj, http://127.0.0.1:44086/_v/vj, http://127.0.0.1:35231/_v/vj, http://127.0.0.1:46864/_v/vj]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:888)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:46864/_v/vj
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:585)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
   [junit4]    > 	... 5 more
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:480)
   [junit4]    > 	... 9 more
   [junit4]   2> 1977347 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[91E270C9DE67D054]-worker) [n:127.0.0.1:44086__v%2Fvj c:awholynewstresscollection_collection5_1 s:shard5 r:core_node12 x:awholynewstresscollection_collection5_1_shard5_replica3] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_91E270C9DE67D054-001
   [junit4]   2> Dec 12, 2015 5:31:30 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=4892, maxDocsPerChunk=810, blockSize=4), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=4892, blockSize=4)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ko_KR, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=92832704,total=477102080
   [junit4]   2> NOTE: All tests run in this JVM: [MBeansHandlerTest, ZkCLITest, ChaosMonkeySafeLeaderTest, OutOfBoxZkACLAndCredentialsProvidersTest, UpdateRequestProcessorFactoryTest, QueryResultKeyTest, TestQueryUtils, SpellPossibilityIteratorTest, TestCollapseQParserPlugin, TestFastLRUCache, TestDFRSimilarityFactory, DocumentBuilderTest, TestSolrCloudWithKerberosAlt, DeleteLastCustomShardedReplicaTest, ChaosMonkeyNothingIsSafeTest, TestManagedResource, SortSpecParsingTest, DirectSolrSpellCheckerTest, TestLMJelinekMercerSimilarityFactory, DocExpirationUpdateProcessorFactoryTest, ClusterStateUpdateTest, TestTrie, TestFileDictionaryLookup, SpellCheckCollatorTest, TestQueryTypes, FastVectorHighlighterTest, HdfsDirectoryTest, TestFieldTypeResource, ConnectionManagerTest, SolrInfoMBeanTest, SchemaVersionSpecificBehaviorTest, HdfsThreadLeakTest, TestQuerySenderListener, DistribCursorPagingTest, TermsComponentTest, WordBreakSolrSpellCheckerTest, TimeZoneUtilsTest, ConvertedLegacyTest, StressHdfsTest, TestReplicationHandlerBackup, TestLFUCache, DistributedFacetPivotSmallAdvancedTest, TestDistribDocBasedVersion, NotRequiredUniqueKeyTest, TestMiniSolrCloudClusterBase, TestScoreJoinQPNoScore, TestTrieFacet, SolrIndexConfigTest, ExplicitHLLTest, SuggesterFSTTest, TestSchemaResource, RecoveryAfterSoftCommitTest, SolrPluginUtilsTest, TestReRankQParserPlugin, TestExactStatsCache, TestMaxScoreQueryParser, TestOrdValues, SolrXmlInZkTest, SharedFSAutoReplicaFailoverUtilsTest, BlockCacheTest, TestOverriddenPrefixQueryForCustomFieldType, TestSchemaManager, FileUtilsTest, TestCloudSchemaless, TestFieldSortValues, MultiTermTest, AssignTest, OpenExchangeRatesOrgProviderTest, ZkSolrClientTest, TestCodecSupport, TestFieldCollectionResource, BasicFunctionalityTest, TestChildDocTransformer, TestSimpleTrackingShardHandler, TestCursorMarkWithoutUniqueKey, DistributedTermsComponentTest, CloudMLTQParserTest, SoftAutoCommitTest, ShardRoutingTest, AddSchemaFieldsUpdateProcessorFactoryTest, DistanceFunctionTest, DocValuesMissingTest, TestBulkSchemaConcurrent, TestCloudPivotFacet, DistributedMLTComponentTest, ConjunctionSolrSpellCheckerTest, SimplePostToolTest, TestRandomForceMergePolicy, TermVectorComponentTest, TestBinaryResponseWriter, BasicDistributedZkTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed [239/555 (2!)] on J1 in 532.01s, 1 test, 1 error <<< FAILURES!

[...truncated 1008 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J1-20151212_045830_148.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/heapdumps/java_pid7675.hprof ...
   [junit4] Heap dump file created [613583618 bytes in 8.932 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 23 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1010: There were test failures: 555 suites (7 ignored), 2192 tests, 1 error, 1 failure, 33 ignored (18 assumptions) [seed: 91E270C9DE67D054]

Total time: 225 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1043 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1043/

2 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1449825644507,generation=2,filelist=[_2njz.fdt, _2njz.fdx, _2njz.fnm, _2njz.nvd, _2njz.nvm, _2njz.si, _2njz_MockRandom_0.doc, _2njz_MockRandom_0.sd, _2njz_MockRandom_0.tbk, _2njz_MockRandom_0.tix, _2nve.cfe, _2nve.cfs, _2nve.si, _2nvh.fdt, _2nvh.fdx, _2nvh.fnm, _2nvh.nvd, _2nvh.nvm, _2nvh.si, _2nvh_MockRandom_0.doc, _2nvh_MockRandom_0.sd, _2nvh_MockRandom_0.tib, _2nvh_MockRandom_0.tii, _2nvn.cfe, _2nvn.cfs, _2nvn.si, _2nvp.fdt, _2nvp.fdx, _2nvp.fnm, _2nvp.nvd, _2nvp.nvm, _2nvp.si, _2nvp_MockRandom_0.doc, _2nvp_MockRandom_0.sd, _2nvp_MockRandom_0.tim, _2nvp_MockRandom_0.tip, _2nvq.cfe, _2nvq.cfs, _2nvq.si, _2nvr.fdt, _2nvr.fdx, _2nvr.fnm, _2nvr.nvd, _2nvr.nvm, _2nvr.si, _2nvr_MockRandom_0.doc, _2nvr_MockRandom_0.sd, _2nvr_MockRandom_0.tbk, _2nvr_MockRandom_0.tix, _2nwm.fdt, _2nwm.fdx, _2nwm.fnm, _2nwm.nvd, _2nwm.nvm, _2nwm.si, _2nwm_MockRandom_0.doc, _2nwm_MockRandom_0.sd, _2nwm_MockRandom_0.tmp, _2nwp.cfe, _2nwp.cfs, _2nwp.si, _2nwq.fdt, _2nwq.fdx, _2nwq.fnm, _2nwq.nvd, _2nwq.nvm, _2nwq.si, _2nwq_MockRandom_0.doc, _2nwq_MockRandom_0.sd, _2nwq_MockRandom_0.tbk, _2nwq_MockRandom_0.tix, _2nwr.fdt, _2nwr.fdx, _2nwr.fnm, _2nwr.nvd, _2nwr.nvm, _2nwr.si, _2nwr_MockRandom_0.doc, _2nwr_MockRandom_0.sd, _2nwr_MockRandom_0.tio, _2nwr_MockRandom_0.tipo, _2nws.fdt, _2nws.fdx, _2nws.fnm, _2nws.nvd, _2nws.nvm, _2nws.si, _2nws_MockRandom_0.doc, _2nws_MockRandom_0.sd, _2nws_MockRandom_0.tio, _2nws_MockRandom_0.tipo, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nwx.fdt, _2nwx.fdx, _2nwx.fnm, _2nwx.nvd, _2nwx.nvm, _2nwx.si, _2nwx_MockRandom_0.doc, _2nwx_MockRandom_0.sd, _2nwx_MockRandom_0.tio, _2nwx_MockRandom_0.tipo, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx3.cfe, _2nx3.cfs, _2nx3.si, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nx5.fdt, _2nx5.fdx, _2nx5.fnm, _2nx5.nvd, _2nx5.nvm, _2nx5.si, _2nx5_MockRandom_0.doc, _2nx5_MockRandom_0.sd, _2nx5_MockRandom_0.tmp, _2nx6.fdt, _2nx6.fdx, _2nx6.fnm, _2nx6.nvd, _2nx6.nvm, _2nx6.si, _2nx6_MockRandom_0.doc, _2nx6_MockRandom_0.sd, _2nx6_MockRandom_0.tib, _2nx6_MockRandom_0.tii, _2nx7.fdt, _2nx7.fdx, _2nx7.fnm, _2nx7.nvd, _2nx7.nvm, _2nx7.si, _2nx7_MockRandom_0.doc, _2nx7_MockRandom_0.sd, _2nx7_MockRandom_0.tbk, _2nx7_MockRandom_0.tix, _2nx8.fdt, _2nx8.fdx, _2nx8.fnm, _2nx8.nvd, _2nx8.nvm, _2nx8.si, _2nx8_MockRandom_0.doc, _2nx8_MockRandom_0.sd, _2nx8_MockRandom_0.tbk, _2nx8_MockRandom_0.tix, _2nx9.cfe, _2nx9.cfs, _2nx9.si, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxd.fdt, _2nxd.fdx, _2nxd.fnm, _2nxd.nvd, _2nxd.nvm, _2nxd.si, _2nxd_MockRandom_0.doc, _2nxd_MockRandom_0.sd, _2nxd_MockRandom_0.tim, _2nxd_MockRandom_0.tip, _2nxe.fdt, _2nxe.fdx, _2nxe.fnm, _2nxe.nvd, _2nxe.nvm, _2nxe.si, _2nxe_MockRandom_0.doc, _2nxe_MockRandom_0.sd, _2nxe_MockRandom_0.tbk, _2nxe_MockRandom_0.tix, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_2]}]> but was:<[{indexVersion=1449825644507,generation=2,filelist=[_2njz.fdt, _2njz.fdx, _2njz.fnm, _2njz.nvd, _2njz.nvm, _2njz.si, _2njz_MockRandom_0.doc, _2njz_MockRandom_0.sd, _2njz_MockRandom_0.tbk, _2njz_MockRandom_0.tix, _2nve.cfe, _2nve.cfs, _2nve.si, _2nvh.fdt, _2nvh.fdx, _2nvh.fnm, _2nvh.nvd, _2nvh.nvm, _2nvh.si, _2nvh_MockRandom_0.doc, _2nvh_MockRandom_0.sd, _2nvh_MockRandom_0.tib, _2nvh_MockRandom_0.tii, _2nvn.cfe, _2nvn.cfs, _2nvn.si, _2nvp.fdt, _2nvp.fdx, _2nvp.fnm, _2nvp.nvd, _2nvp.nvm, _2nvp.si, _2nvp_MockRandom_0.doc, _2nvp_MockRandom_0.sd, _2nvp_MockRandom_0.tim, _2nvp_MockRandom_0.tip, _2nvq.cfe, _2nvq.cfs, _2nvq.si, _2nvr.fdt, _2nvr.fdx, _2nvr.fnm, _2nvr.nvd, _2nvr.nvm, _2nvr.si, _2nvr_MockRandom_0.doc, _2nvr_MockRandom_0.sd, _2nvr_MockRandom_0.tbk, _2nvr_MockRandom_0.tix, _2nwm.fdt, _2nwm.fdx, _2nwm.fnm, _2nwm.nvd, _2nwm.nvm, _2nwm.si, _2nwm_MockRandom_0.doc, _2nwm_MockRandom_0.sd, _2nwm_MockRandom_0.tmp, _2nwp.cfe, _2nwp.cfs, _2nwp.si, _2nwq.fdt, _2nwq.fdx, _2nwq.fnm, _2nwq.nvd, _2nwq.nvm, _2nwq.si, _2nwq_MockRandom_0.doc, _2nwq_MockRandom_0.sd, _2nwq_MockRandom_0.tbk, _2nwq_MockRandom_0.tix, _2nwr.fdt, _2nwr.fdx, _2nwr.fnm, _2nwr.nvd, _2nwr.nvm, _2nwr.si, _2nwr_MockRandom_0.doc, _2nwr_MockRandom_0.sd, _2nwr_MockRandom_0.tio, _2nwr_MockRandom_0.tipo, _2nws.fdt, _2nws.fdx, _2nws.fnm, _2nws.nvd, _2nws.nvm, _2nws.si, _2nws_MockRandom_0.doc, _2nws_MockRandom_0.sd, _2nws_MockRandom_0.tio, _2nws_MockRandom_0.tipo, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nwx.fdt, _2nwx.fdx, _2nwx.fnm, _2nwx.nvd, _2nwx.nvm, _2nwx.si, _2nwx_MockRandom_0.doc, _2nwx_MockRandom_0.sd, _2nwx_MockRandom_0.tio, _2nwx_MockRandom_0.tipo, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx3.cfe, _2nx3.cfs, _2nx3.si, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nx5.fdt, _2nx5.fdx, _2nx5.fnm, _2nx5.nvd, _2nx5.nvm, _2nx5.si, _2nx5_MockRandom_0.doc, _2nx5_MockRandom_0.sd, _2nx5_MockRandom_0.tmp, _2nx6.fdt, _2nx6.fdx, _2nx6.fnm, _2nx6.nvd, _2nx6.nvm, _2nx6.si, _2nx6_MockRandom_0.doc, _2nx6_MockRandom_0.sd, _2nx6_MockRandom_0.tib, _2nx6_MockRandom_0.tii, _2nx7.fdt, _2nx7.fdx, _2nx7.fnm, _2nx7.nvd, _2nx7.nvm, _2nx7.si, _2nx7_MockRandom_0.doc, _2nx7_MockRandom_0.sd, _2nx7_MockRandom_0.tbk, _2nx7_MockRandom_0.tix, _2nx8.fdt, _2nx8.fdx, _2nx8.fnm, _2nx8.nvd, _2nx8.nvm, _2nx8.si, _2nx8_MockRandom_0.doc, _2nx8_MockRandom_0.sd, _2nx8_MockRandom_0.tbk, _2nx8_MockRandom_0.tix, _2nx9.cfe, _2nx9.cfs, _2nx9.si, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxd.fdt, _2nxd.fdx, _2nxd.fnm, _2nxd.nvd, _2nxd.nvm, _2nxd.si, _2nxd_MockRandom_0.doc, _2nxd_MockRandom_0.sd, _2nxd_MockRandom_0.tim, _2nxd_MockRandom_0.tip, _2nxe.fdt, _2nxe.fdx, _2nxe.fnm, _2nxe.nvd, _2nxe.nvm, _2nxe.si, _2nxe_MockRandom_0.doc, _2nxe_MockRandom_0.sd, _2nxe_MockRandom_0.tbk, _2nxe_MockRandom_0.tix, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_2]}, {indexVersion=1449825644507,generation=3,filelist=[_2nvt.fdt, _2nvt.fdx, _2nvt.fnm, _2nvt.nvd, _2nvt.nvm, _2nvt.si, _2nvt_MockRandom_0.doc, _2nvt_MockRandom_0.sd, _2nvt_MockRandom_0.tmp, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxg.fdt, _2nxg.fdx, _2nxg.fnm, _2nxg.nvd, _2nxg.nvm, _2nxg.si, _2nxg_MockRandom_0.doc, _2nxg_MockRandom_0.sd, _2nxg_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1449825644507,generation=2,filelist=[_2njz.fdt, _2njz.fdx, _2njz.fnm, _2njz.nvd, _2njz.nvm, _2njz.si, _2njz_MockRandom_0.doc, _2njz_MockRandom_0.sd, _2njz_MockRandom_0.tbk, _2njz_MockRandom_0.tix, _2nve.cfe, _2nve.cfs, _2nve.si, _2nvh.fdt, _2nvh.fdx, _2nvh.fnm, _2nvh.nvd, _2nvh.nvm, _2nvh.si, _2nvh_MockRandom_0.doc, _2nvh_MockRandom_0.sd, _2nvh_MockRandom_0.tib, _2nvh_MockRandom_0.tii, _2nvn.cfe, _2nvn.cfs, _2nvn.si, _2nvp.fdt, _2nvp.fdx, _2nvp.fnm, _2nvp.nvd, _2nvp.nvm, _2nvp.si, _2nvp_MockRandom_0.doc, _2nvp_MockRandom_0.sd, _2nvp_MockRandom_0.tim, _2nvp_MockRandom_0.tip, _2nvq.cfe, _2nvq.cfs, _2nvq.si, _2nvr.fdt, _2nvr.fdx, _2nvr.fnm, _2nvr.nvd, _2nvr.nvm, _2nvr.si, _2nvr_MockRandom_0.doc, _2nvr_MockRandom_0.sd, _2nvr_MockRandom_0.tbk, _2nvr_MockRandom_0.tix, _2nwm.fdt, _2nwm.fdx, _2nwm.fnm, _2nwm.nvd, _2nwm.nvm, _2nwm.si, _2nwm_MockRandom_0.doc, _2nwm_MockRandom_0.sd, _2nwm_MockRandom_0.tmp, _2nwp.cfe, _2nwp.cfs, _2nwp.si, _2nwq.fdt, _2nwq.fdx, _2nwq.fnm, _2nwq.nvd, _2nwq.nvm, _2nwq.si, _2nwq_MockRandom_0.doc, _2nwq_MockRandom_0.sd, _2nwq_MockRandom_0.tbk, _2nwq_MockRandom_0.tix, _2nwr.fdt, _2nwr.fdx, _2nwr.fnm, _2nwr.nvd, _2nwr.nvm, _2nwr.si, _2nwr_MockRandom_0.doc, _2nwr_MockRandom_0.sd, _2nwr_MockRandom_0.tio, _2nwr_MockRandom_0.tipo, _2nws.fdt, _2nws.fdx, _2nws.fnm, _2nws.nvd, _2nws.nvm, _2nws.si, _2nws_MockRandom_0.doc, _2nws_MockRandom_0.sd, _2nws_MockRandom_0.tio, _2nws_MockRandom_0.tipo, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nwx.fdt, _2nwx.fdx, _2nwx.fnm, _2nwx.nvd, _2nwx.nvm, _2nwx.si, _2nwx_MockRandom_0.doc, _2nwx_MockRandom_0.sd, _2nwx_MockRandom_0.tio, _2nwx_MockRandom_0.tipo, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx3.cfe, _2nx3.cfs, _2nx3.si, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nx5.fdt, _2nx5.fdx, _2nx5.fnm, _2nx5.nvd, _2nx5.nvm, _2nx5.si, _2nx5_MockRandom_0.doc, _2nx5_MockRandom_0.sd, _2nx5_MockRandom_0.tmp, _2nx6.fdt, _2nx6.fdx, _2nx6.fnm, _2nx6.nvd, _2nx6.nvm, _2nx6.si, _2nx6_MockRandom_0.doc, _2nx6_MockRandom_0.sd, _2nx6_MockRandom_0.tib, _2nx6_MockRandom_0.tii, _2nx7.fdt, _2nx7.fdx, _2nx7.fnm, _2nx7.nvd, _2nx7.nvm, _2nx7.si, _2nx7_MockRandom_0.doc, _2nx7_MockRandom_0.sd, _2nx7_MockRandom_0.tbk, _2nx7_MockRandom_0.tix, _2nx8.fdt, _2nx8.fdx, _2nx8.fnm, _2nx8.nvd, _2nx8.nvm, _2nx8.si, _2nx8_MockRandom_0.doc, _2nx8_MockRandom_0.sd, _2nx8_MockRandom_0.tbk, _2nx8_MockRandom_0.tix, _2nx9.cfe, _2nx9.cfs, _2nx9.si, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxd.fdt, _2nxd.fdx, _2nxd.fnm, _2nxd.nvd, _2nxd.nvm, _2nxd.si, _2nxd_MockRandom_0.doc, _2nxd_MockRandom_0.sd, _2nxd_MockRandom_0.tim, _2nxd_MockRandom_0.tip, _2nxe.fdt, _2nxe.fdx, _2nxe.fnm, _2nxe.nvd, _2nxe.nvm, _2nxe.si, _2nxe_MockRandom_0.doc, _2nxe_MockRandom_0.sd, _2nxe_MockRandom_0.tbk, _2nxe_MockRandom_0.tix, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_2]}]> but was:<[{indexVersion=1449825644507,generation=2,filelist=[_2njz.fdt, _2njz.fdx, _2njz.fnm, _2njz.nvd, _2njz.nvm, _2njz.si, _2njz_MockRandom_0.doc, _2njz_MockRandom_0.sd, _2njz_MockRandom_0.tbk, _2njz_MockRandom_0.tix, _2nve.cfe, _2nve.cfs, _2nve.si, _2nvh.fdt, _2nvh.fdx, _2nvh.fnm, _2nvh.nvd, _2nvh.nvm, _2nvh.si, _2nvh_MockRandom_0.doc, _2nvh_MockRandom_0.sd, _2nvh_MockRandom_0.tib, _2nvh_MockRandom_0.tii, _2nvn.cfe, _2nvn.cfs, _2nvn.si, _2nvp.fdt, _2nvp.fdx, _2nvp.fnm, _2nvp.nvd, _2nvp.nvm, _2nvp.si, _2nvp_MockRandom_0.doc, _2nvp_MockRandom_0.sd, _2nvp_MockRandom_0.tim, _2nvp_MockRandom_0.tip, _2nvq.cfe, _2nvq.cfs, _2nvq.si, _2nvr.fdt, _2nvr.fdx, _2nvr.fnm, _2nvr.nvd, _2nvr.nvm, _2nvr.si, _2nvr_MockRandom_0.doc, _2nvr_MockRandom_0.sd, _2nvr_MockRandom_0.tbk, _2nvr_MockRandom_0.tix, _2nwm.fdt, _2nwm.fdx, _2nwm.fnm, _2nwm.nvd, _2nwm.nvm, _2nwm.si, _2nwm_MockRandom_0.doc, _2nwm_MockRandom_0.sd, _2nwm_MockRandom_0.tmp, _2nwp.cfe, _2nwp.cfs, _2nwp.si, _2nwq.fdt, _2nwq.fdx, _2nwq.fnm, _2nwq.nvd, _2nwq.nvm, _2nwq.si, _2nwq_MockRandom_0.doc, _2nwq_MockRandom_0.sd, _2nwq_MockRandom_0.tbk, _2nwq_MockRandom_0.tix, _2nwr.fdt, _2nwr.fdx, _2nwr.fnm, _2nwr.nvd, _2nwr.nvm, _2nwr.si, _2nwr_MockRandom_0.doc, _2nwr_MockRandom_0.sd, _2nwr_MockRandom_0.tio, _2nwr_MockRandom_0.tipo, _2nws.fdt, _2nws.fdx, _2nws.fnm, _2nws.nvd, _2nws.nvm, _2nws.si, _2nws_MockRandom_0.doc, _2nws_MockRandom_0.sd, _2nws_MockRandom_0.tio, _2nws_MockRandom_0.tipo, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nwx.fdt, _2nwx.fdx, _2nwx.fnm, _2nwx.nvd, _2nwx.nvm, _2nwx.si, _2nwx_MockRandom_0.doc, _2nwx_MockRandom_0.sd, _2nwx_MockRandom_0.tio, _2nwx_MockRandom_0.tipo, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx3.cfe, _2nx3.cfs, _2nx3.si, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nx5.fdt, _2nx5.fdx, _2nx5.fnm, _2nx5.nvd, _2nx5.nvm, _2nx5.si, _2nx5_MockRandom_0.doc, _2nx5_MockRandom_0.sd, _2nx5_MockRandom_0.tmp, _2nx6.fdt, _2nx6.fdx, _2nx6.fnm, _2nx6.nvd, _2nx6.nvm, _2nx6.si, _2nx6_MockRandom_0.doc, _2nx6_MockRandom_0.sd, _2nx6_MockRandom_0.tib, _2nx6_MockRandom_0.tii, _2nx7.fdt, _2nx7.fdx, _2nx7.fnm, _2nx7.nvd, _2nx7.nvm, _2nx7.si, _2nx7_MockRandom_0.doc, _2nx7_MockRandom_0.sd, _2nx7_MockRandom_0.tbk, _2nx7_MockRandom_0.tix, _2nx8.fdt, _2nx8.fdx, _2nx8.fnm, _2nx8.nvd, _2nx8.nvm, _2nx8.si, _2nx8_MockRandom_0.doc, _2nx8_MockRandom_0.sd, _2nx8_MockRandom_0.tbk, _2nx8_MockRandom_0.tix, _2nx9.cfe, _2nx9.cfs, _2nx9.si, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxd.fdt, _2nxd.fdx, _2nxd.fnm, _2nxd.nvd, _2nxd.nvm, _2nxd.si, _2nxd_MockRandom_0.doc, _2nxd_MockRandom_0.sd, _2nxd_MockRandom_0.tim, _2nxd_MockRandom_0.tip, _2nxe.fdt, _2nxe.fdx, _2nxe.fnm, _2nxe.nvd, _2nxe.nvm, _2nxe.si, _2nxe_MockRandom_0.doc, _2nxe_MockRandom_0.sd, _2nxe_MockRandom_0.tbk, _2nxe_MockRandom_0.tix, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_2]}, {indexVersion=1449825644507,generation=3,filelist=[_2nvt.fdt, _2nvt.fdx, _2nvt.fnm, _2nvt.nvd, _2nvt.nvm, _2nvt.si, _2nvt_MockRandom_0.doc, _2nvt_MockRandom_0.sd, _2nvt_MockRandom_0.tmp, _2nwu.fdt, _2nwu.fdx, _2nwu.fnm, _2nwu.nvd, _2nwu.nvm, _2nwu.si, _2nwu_MockRandom_0.doc, _2nwu_MockRandom_0.sd, _2nwu_MockRandom_0.tim, _2nwu_MockRandom_0.tip, _2nx1.fdt, _2nx1.fdx, _2nx1.fnm, _2nx1.nvd, _2nx1.nvm, _2nx1.si, _2nx1_MockRandom_0.doc, _2nx1_MockRandom_0.sd, _2nx1_MockRandom_0.tmp, _2nx2.fdt, _2nx2.fdx, _2nx2.fnm, _2nx2.nvd, _2nx2.nvm, _2nx2.si, _2nx2_MockRandom_0.doc, _2nx2_MockRandom_0.sd, _2nx2_MockRandom_0.tib, _2nx2_MockRandom_0.tiv, _2nx4.fdt, _2nx4.fdx, _2nx4.fnm, _2nx4.nvd, _2nx4.nvm, _2nx4.si, _2nx4_MockRandom_0.doc, _2nx4_MockRandom_0.sd, _2nx4_MockRandom_0.tbk, _2nx4_MockRandom_0.tix, _2nxa.fdt, _2nxa.fdx, _2nxa.fnm, _2nxa.nvd, _2nxa.nvm, _2nxa.si, _2nxa_MockRandom_0.doc, _2nxa_MockRandom_0.sd, _2nxa_MockRandom_0.tio, _2nxa_MockRandom_0.tipo, _2nxb.fdt, _2nxb.fdx, _2nxb.fnm, _2nxb.nvd, _2nxb.nvm, _2nxb.si, _2nxb_MockRandom_0.doc, _2nxb_MockRandom_0.sd, _2nxb_MockRandom_0.tmp, _2nxc.fdt, _2nxc.fdx, _2nxc.fnm, _2nxc.nvd, _2nxc.nvm, _2nxc.si, _2nxc_MockRandom_0.doc, _2nxc_MockRandom_0.sd, _2nxc_MockRandom_0.tio, _2nxc_MockRandom_0.tipo, _2nxf.fdt, _2nxf.fdx, _2nxf.fnm, _2nxf.nvd, _2nxf.nvm, _2nxf.si, _2nxf_MockRandom_0.doc, _2nxf_MockRandom_0.sd, _2nxf_MockRandom_0.tmp, _2nxg.fdt, _2nxg.fdx, _2nxg.fnm, _2nxg.nvd, _2nxg.nvm, _2nxg.si, _2nxg_MockRandom_0.doc, _2nxg_MockRandom_0.sd, _2nxg_MockRandom_0.tmp, _2nxh.fdt, _2nxh.fdx, _2nxh.fnm, _2nxh.nvd, _2nxh.nvm, _2nxh.si, _2nxh_MockRandom_0.doc, _2nxh_MockRandom_0.sd, _2nxh_MockRandom_0.tim, _2nxh_MockRandom_0.tip, _2nxi.fdt, _2nxi.fdx, _2nxi.fnm, _2nxi.nvd, _2nxi.nvm, _2nxi.si, _2nxi_MockRandom_0.doc, _2nxi_MockRandom_0.sd, _2nxi_MockRandom_0.tim, _2nxi_MockRandom_0.tip, _2nxj.fdt, _2nxj.fdx, _2nxj.fnm, _2nxj.nvd, _2nxj.nvm, _2nxj.si, _2nxj_MockRandom_0.doc, _2nxj_MockRandom_0.sd, _2nxj_MockRandom_0.tib, _2nxj_MockRandom_0.tii, _2nxk.fdt, _2nxk.fdx, _2nxk.fnm, _2nxk.nvd, _2nxk.nvm, _2nxk.si, _2nxk_MockRandom_0.doc, _2nxk_MockRandom_0.sd, _2nxk_MockRandom_0.tim, _2nxk_MockRandom_0.tip, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([252FCDCEA534147F:F8D6FED57C1A7C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Fri Dec 11 22:22:02 GMT+13:00 2015

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Fri Dec 11 22:22:02 GMT+13:00 2015
	at __randomizedtesting.SeedInfo.seed([252FCDCEA534147F:FE84CD08A01C7DCC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1419)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:771)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10887 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/init-core-data-001
   [junit4]   2> 1107487 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestIndexAndConfigReplication
   [junit4]   2> 1107487 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1
   [junit4]   2> 1107491 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1107493 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16de7d5f{/solr,null,AVAILABLE}
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.ServerConnector Started ServerConnector@68f8f3b3{HTTP/1.1}{127.0.0.1:35053}
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.Server Started @1110494ms
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=35053, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1/data, hostContext=/solr}
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001'
   [junit4]   2> 1107494 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1107495 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1107495 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/solr.xml
   [junit4]   2> 1107510 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/.
   [junit4]   2> 1107510 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer New CoreContainer 1347987264
   [junit4]   2> 1107510 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001]
   [junit4]   2> 1107510 WARN  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/lib
   [junit4]   2> 1107510 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1107511 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1107512 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1107512 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1107512 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1107512 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1107513 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/.
   [junit4]   2> 1107514 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {transient=false, loadOnStartup=true, config=solrconfig.xml, configSetProperties=configsetprops.json, name=collection1, dataDir=data/, schema=schema.xml}
   [junit4]   2> 1107514 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1
   [junit4]   2> 1107514 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1107515 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1'
   [junit4]   2> 1107515 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1107515 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1107515 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1107515 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1107523 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1107536 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1107561 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1107565 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1107573 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1'
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data
   [junit4]   2> 1107575 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 1107576 WARN  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1107576 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 1107576 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1187930328, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1107577 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@49765f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6fa6fbd4),segFN=segments_1,generation=1}
   [junit4]   2> 1107577 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1107577 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/./collection1/data/
   [junit4]   2> 1107578 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1107581 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1107583 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1107584 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1107585 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1107601 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1107601 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/threads,/schema,/update/json/docs,/admin/system,standard,lazy,/admin/logging,/admin/properties,/update/json,/admin/plugins,/get,/replication,/admin/file,/admin/mbeans,defaults,/admin/segments,/update/csv,/update,/admin/luke,/config,/admin/ping
   [junit4]   2> 1107602 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@40455297
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@49765f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6fa6fbd4),segFN=segments_1,generation=1}
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1107603 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@37bf255[collection1] main
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1/conf
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-001/collection1/conf
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1107604 INFO  (coreLoadExecutor-2297-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1107604 INFO  (searcherExecutor-2298-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@37bf255[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1108016 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1
   [junit4]   2> 1108020 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1108022 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62dcc762{/solr,null,AVAILABLE}
   [junit4]   2> 1108022 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6ff0f6d1{HTTP/1.1}{127.0.0.1:46287}
   [junit4]   2> 1108022 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.Server Started @1111022ms
   [junit4]   2> 1108022 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=46287}
   [junit4]   2> 1108023 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 1108023 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002'
   [junit4]   2> 1108023 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1108023 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1108023 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/solr.xml
   [junit4]   2> 1108037 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/.
   [junit4]   2> 1108037 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer New CoreContainer 974364157
   [junit4]   2> 1108037 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002]
   [junit4]   2> 1108037 WARN  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/lib
   [junit4]   2> 1108037 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1108039 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1108039 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1108039 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1108039 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1108039 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1108040 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/.
   [junit4]   2> 1108041 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {transient=false, dataDir=data/, config=solrconfig.xml, loadOnStartup=true, configSetProperties=configsetprops.json, name=collection1, schema=schema.xml}
   [junit4]   2> 1108041 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1
   [junit4]   2> 1108041 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1108042 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1'
   [junit4]   2> 1108042 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1108042 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1108042 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1108042 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1108050 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1108063 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1108089 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1108094 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1108101 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1108103 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1'
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data
   [junit4]   2> 1108104 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 1108105 WARN  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1108105 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1108105 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1187930328, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1108106 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2a08ff5b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@34b4c4ec),segFN=segments_1,generation=1}
   [junit4]   2> 1108106 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1108106 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/./collection1/data/
   [junit4]   2> 1108108 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1108111 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1108112 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1108113 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1108114 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1108116 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1108116 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/threads,/schema,/update/json/docs,/admin/system,standard,lazy,/admin/logging,/admin/properties,/update/json,/admin/plugins,/get,/replication,/admin/file,/admin/mbeans,defaults,/admin/segments,/update/csv,/update,/admin/luke,/config,/admin/ping
   [junit4]   2> 1108117 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7155e71f
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2a08ff5b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@34b4c4ec),segFN=segments_1,generation=1}
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@41d39a0e[collection1] main
   [junit4]   2> 1108118 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1/conf
   [junit4]   2> 1108119 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1108119 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-002/collection1/conf
   [junit4]   2> 1108119 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1108121 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1108121 INFO  (coreLoadExecutor-2307-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1108121 INFO  (searcherExecutor-2308-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@41d39a0e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1108124 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={wt=javabin&qt=/replication&command=indexversion&version=2} status=0 QTime=0 
   [junit4]   2> 1108545 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/select params={wt=javabin&version=2&sort=id+desc&q=*:*} hits=0 status=0 QTime=0 
   [junit4]   2> 1108549 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 1
   [junit4]   2> 1108553 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 1
   [junit4]   2> 1108556 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 1
   [junit4]   2> 1108559 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
   [junit4]   2> 1108562 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 0
   [junit4]   2> 1108566 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 1
   [junit4]   2> 1108568 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 0
   [junit4]   2> 1108597 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 1
   [junit4]   2> 1108599 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 0
   [junit4]   2> 1108602 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 1
   [junit4]   2> 1108605 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 0
   [junit4]   2> 1108607 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 1
   [junit4]   2> 1108610 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 0
   [junit4]   2> 1108613 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 1
   [junit4]   2> 1108615 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 0
   [junit4]   2> 1108618 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 1
   [junit4]   2> 1108620 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 0
   [junit4]   2> 1108623 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 1
   [junit4]   2> 1108626 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 0
   [junit4]   2> 1108629 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 1
   [junit4]   2> 1108631 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 0
   [junit4]   2> 1108634 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 1
   [junit4]   2> 1108637 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 0
   [junit4]   2> 1108640 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 1
   [junit4]   2> 1108643 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 0
   [junit4]   2> 1108645 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 1
   [junit4]   2> 1108647 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26]} 0 0
   [junit4]   2> 1108650 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27]} 0 1
   [junit4]   2> 1108652 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[28]} 0 0
   [junit4]   2> 1108655 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[29]} 0 1
   [junit4]   2> 1108657 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[30]} 0 0
   [junit4]   2> 1108660 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[31]} 0 1
   [junit4]   2> 1108662 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[32]} 0 0
   [junit4]   2> 1108665 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[33]} 0 1
   [junit4]   2> 1108667 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[34]} 0 0
   [junit4]   2> 1108670 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[35]} 0 1
   [junit4]   2> 1108672 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36]} 0 0
   [junit4]   2> 1108675 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[37]} 0 1
   [junit4]   2> 1108677 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[38]} 0 0
   [junit4]   2> 1108680 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39]} 0 1
   [junit4]   2> 1108682 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[40]} 0 0
   [junit4]   2> 1108685 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41]} 0 1
   [junit4]   2> 1108687 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[42]} 0 0
   [junit4]   2> 1108695 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[43]} 0 5
   [junit4]   2> 1108697 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44]} 0 0
   [junit4]   2> 1108700 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45]} 0 1
   [junit4]   2> 1108702 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[46]} 0 0
   [junit4]   2> 1108705 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47]} 0 1
   [junit4]   2> 1108708 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48]} 0 0
   [junit4]   2> 1108711 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[49]} 0 1
   [junit4]   2> 1108713 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[50]} 0 0
   [junit4]   2> 1108717 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[51]} 0 1
   [junit4]   2> 1108719 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[52]} 0 0
   [junit4]   2> 1108722 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[53]} 0 1
   [junit4]   2> 1108724 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54]} 0 0
   [junit4]   2> 1108727 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[55]} 0 1
   [junit4]   2> 1108729 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56]} 0 0
   [junit4]   2> 1108732 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[57]} 0 1
   [junit4]   2> 1108734 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[58]} 0 0
   [junit4]   2> 1108738 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[59]} 0 1
   [junit4]   2> 1108740 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[60]} 0 0
   [junit4]   2> 1108803 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[61]} 0 61
   [junit4]   2> 1108806 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[62]} 0 0
   [junit4]   2> 1108812 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63]} 0 4
   [junit4]   2> 1108814 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64]} 0 0
   [junit4]   2> 1108818 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[65]} 0 2
   [junit4]   2> 1108820 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66]} 0 0
   [junit4]   2> 1108824 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[67]} 0 1
   [junit4]   2> 1108826 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[68]} 0 0
   [junit4]   2> 1108829 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[69]} 0 1
   [junit4]   2> 1108831 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[70]} 0 0
   [junit4]   2> 1108837 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[71]} 0 4
   [junit4]   2> 1108840 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[72]} 0 0
   [junit4]   2> 1108842 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[73]} 0 1
   [junit4]   2> 1108845 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[74]} 0 0
   [junit4]   2> 1108851 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[75]} 0 3
   [junit4]   2> 1108853 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[76]} 0 0
   [junit4]   2> 1108856 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[77]} 0 1
   [junit4]   2> 1108858 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[78]} 0 0
   [junit4]   2> 1108861 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[79]} 0 1
   [junit4]   2> 1108863 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[80]} 0 0
   [junit4]   2> 1108866 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[81]} 0 1
   [junit4]   2> 1108868 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[82]} 0 0
   [junit4]   2> 1108871 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[83]} 0 1
   [junit4]   2> 1108873 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[84]} 0 0
   [junit4]   2> 1108875 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[85]} 0 1
   [junit4]   2> 1108877 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[86]} 0 0
   [junit4]   2> 1108880 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[87]} 0 1
   [junit4]   2> 1108882 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[88]} 0 0
   [junit4]   2> 1108885 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[89]} 0 1
   [junit4]   2> 1108887 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[90]} 0 0
   [junit4]   2> 1108890 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[91]} 0 1
   [junit4]   2> 1108893 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[92]} 0 0
   [junit4]   2> 1108895 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[93]} 0 1
   [junit4]   2> 1108897 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[94]} 0 0
   [junit4]   2> 1108900 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[95]} 0 1
   [junit4]   2> 1108902 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[96]} 0 0
   [junit4]   2> 1108905 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[97]} 0 1
   [junit4]   2> 1108907 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[98]} 0 0
   [junit4]   2> 1108910 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[99]} 0 1
   [junit4]   2> 1108912 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[100]} 0 0
   [junit4]   2> 1108916 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[101]} 0 1
   [junit4]   2> 1108919 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[102]} 0 1
   [junit4]   2> 1108929 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[103]} 0 8
   [junit4]   2> 1108931 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[104]} 0 0
   [junit4]   2> 1108936 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[105]} 0 3
   [junit4]   2> 1108939 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[106]} 0 0
   [junit4]   2> 1108943 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[107]} 0 2
   [junit4]   2> 1108994 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[108]} 0 49
   [junit4]   2> 1108997 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[109]} 0 1
   [junit4]   2> 1108999 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[110]} 0 0
   [junit4]   2> 1109002 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[111]} 0 1
   [junit4]   2> 1109004 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[112]} 0 0
   [junit4]   2> 1109007 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[113]} 0 1
   [junit4]   2> 1109009 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[114]} 0 0
   [junit4]   2> 1109012 INFO  (qtp351150450-56795) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[115]} 0 0
   [junit4]   2> 1109014 INFO  (qtp351150450-56789) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[116]} 0 0
   [junit4]   2> 1109017 INFO  (qtp351150450-56796) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[117]} 0 0
   [junit4]   2> 1109019 INFO  (qtp351150450-56791) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[118]} 0 0
   [junit4]   2> 1109021 INFO  (qtp351150450-56792) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[119]} 0 1
   [junit4]   2> 1109024 INFO  (qtp351150450-56790) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [c

[...truncated too long message...]

dler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-027/./collection1/data
   [junit4]   2> 2123296 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-027/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-027/./collection1/data/index;done=false>>]
   [junit4]   2> 2123296 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-027/./collection1/data/index
   [junit4]   2> 2123296 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5f966a00{/solr,null,UNAVAILABLE}
   [junit4]   2> 2123297 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@20a391fd{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 2123297 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1069535444
   [junit4]   2> 2123298 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@39384d4e
   [junit4]   2> 2123298 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 2123298 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data/index.20151211222255571 [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data/index.20151211222255571;done=false>>]
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data/index.20151211222255571
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data;done=false>>]
   [junit4]   2> 2123299 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001/solr-instance-028/./collection1/data
   [junit4]   2> 2123300 INFO  (TEST-TestReplicationHandler.doTestIndexAndConfigAliasReplication-seed#[252FCDCEA534147F]) [    x:collection1] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@b548fe8{/solr,null,UNAVAILABLE}
   [junit4]   2> 2123301 INFO  (SUITE-TestReplicationHandler-seed#[252FCDCEA534147F]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_252FCDCEA534147F-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {newname=PostingsFormat(name=MockRandom), name=PostingsFormat(name=MockRandom), id=PostingsFormat(name=MockRandom)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr, timezone=Etc/GMT-13
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=149649120,total=530055168
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolr4Spatial, BasicAuthIntegrationTest, TestValueSourceCache, TestRestManager, TestRandomDVFaceting, TestDocBasedVersionConstraints, UUIDFieldTest, TestSortingResponseWriter, DistributedFacetPivotWhiteBoxTest, DefaultValueUpdateProcessorTest, TestShardHandlerFactory, RankQueryTest, HLLUtilTest, TestRTGBase, TestPKIAuthenticationPlugin, TestSolrQueryParserResource, TestBlendedInfixSuggestions, IndexSchemaTest, VersionInfoTest, TestManagedSchema, RAMDirectoryFactoryTest, BigEndianAscendingWordDeserializerTest, TestCSVLoader, TestNoOpRegenerator, ReturnFieldsTest, TestShortCircuitedRequests, TestMiniSolrCloudCluster, TestFaceting, TestStressLucene, TestHighFrequencyDictionaryFactory, TestRealTimeGet, ResponseHeaderTest, UUIDUpdateProcessorFallbackTest, CopyFieldTest, TestBinaryField, TestFoldingMultitermQuery, TestDistributedSearch, TestReplicaProperties, DistributedSuggestComponentTest, ResponseLogComponentTest, CloudExitableDirectoryReaderTest, StandardRequestHandlerTest, OverseerTaskQueueTest, DeleteShardTest, JsonLoaderTest, DocumentAnalysisRequestHandlerTest, SynonymTokenizerTest, TestLRUCache, OutputWriterTest, TestManagedSynonymFilterFactory, DistributedSpellCheckComponentTest, OverseerCollectionConfigSetProcessorTest, TestJsonRequest, DateFieldTest, TestDocSet, CacheHeaderTest, TestAtomicUpdateErrorCases, PreAnalyzedFieldTest, EchoParamsTest, DisMaxRequestHandlerTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsWriteToMultipleCollectionsTest, AddBlockUpdateTest, ParsingFieldUpdateProcessorsTest, AnalyticsQueryTest, CollectionTooManyReplicasTest, TestStressUserVersions, HdfsDirectoryFactoryTest, DeleteInactiveReplicaTest, InfoHandlerTest, TestReplicationHandler]
   [junit4] Completed [217/554 (1!)] on J0 in 1015.84s, 13 tests, 2 failures <<< FAILURES!

[...truncated 1071 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1010: There were test failures: 554 suites (7 ignored), 2188 tests, 2 failures, 33 ignored (18 assumptions) [seed: 252FCDCEA534147F]

Total time: 215 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1042 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1042/

1 tests failed.
FAILED:  org.apache.lucene.search.TestGeoPointQuery.testRandomBig

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([5DB3A1C37B327030:DAE4DC4CEA6B0CB0]:0)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:427)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
	at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:384)
	at org.apache.lucene.index.BufferedUpdatesStream.openSegmentStates(BufferedUpdatesStream.java:416)
	at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:261)
	at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3161)
	at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3147)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3124)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3087)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1741)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1721)
	at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:596)
	at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
	at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)




Build Log:
[...truncated 8481 lines...]
   [junit4] Suite: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.method=testRandomBig -Dtests.seed=5DB3A1C37B327030 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=el -Dtests.timezone=Etc/Universal -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    126s J1 | TestGeoPointQuery.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5DB3A1C37B327030:DAE4DC4CEA6B0CB0]:0)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:427)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
   [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    > 	at org.apache.lucene.index.BufferedUpdatesStream$SegmentState.<init>(BufferedUpdatesStream.java:384)
   [junit4]    > 	at org.apache.lucene.index.BufferedUpdatesStream.openSegmentStates(BufferedUpdatesStream.java:416)
   [junit4]    > 	at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:261)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3161)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3147)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3124)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3087)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1741)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1721)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:596)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/sandbox/test/J1/temp/lucene.search.TestGeoPointQuery_5DB3A1C37B327030-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {point=PostingsFormat(name=Direct), id=PostingsFormat(name=Memory doPackFST= false), string=Lucene50(blocksize=128)}, docValues:{point=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Memory)}, sim=DefaultSimilarity, locale=el, timezone=Etc/Universal
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=196115464,total=403701760
   [junit4]   2> NOTE: All tests run in this JVM: [DuplicateFilterTest, TestGeoPointQuery]
   [junit4] Completed [18/20 (1!)] on J1 in 386.20s, 23 tests, 1 error <<< FAILURES!

[...truncated 4 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/sandbox/test/temp/junit4-J1-20151210_164328_858.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/heapdumps/java_pid18413.hprof ...
   [junit4] Heap dump file created [552835499 bytes in 5.683 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 13 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2252: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1010: There were test failures: 20 suites, 142 tests, 1 error [seed: 5DB3A1C37B327030]

Total time: 116 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1041 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1041/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.test

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([BF6DB73668EA9BBA:373988ECC616F642]:0)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.getCoreContainer(JettySolrRunner.java:374)
	at org.apache.solr.cloud.ChaosMonkey.expireSession(ChaosMonkey.java:124)
	at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10273 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/init-core-data-001
   [junit4]   2> 703861 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 703950 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 703966 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 703977 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 703992 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_41510_hdfs____.y4dasf/webapp
   [junit4]   2> 704148 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 704475 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41510
   [junit4]   2> 704620 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 704622 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 704640 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40855_datanode____4gus3n/webapp
   [junit4]   2> 704832 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 705128 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40855
   [junit4]   2> 705246 WARN  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 705251 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 705265 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_59784_datanode____.n9srii/webapp
   [junit4]   2> 705465 INFO  (IPC Server handler 3 on 58696) [    ] BlockStateChange BLOCK* processReport: from storage DS-13b02519-4d94-4f73-a6d3-618de83dacac node DatanodeRegistration(127.0.0.1, datanodeUuid=ca1780d1-b226-41a3-926a-10724401377c, infoPort=40855, ipcPort=36581, storageInfo=lv=-56;cid=testClusterID;nsid=911232124;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 705465 INFO  (IPC Server handler 3 on 58696) [    ] BlockStateChange BLOCK* processReport: from storage DS-434609a3-3092-4032-8859-72cd30a2283f node DatanodeRegistration(127.0.0.1, datanodeUuid=ca1780d1-b226-41a3-926a-10724401377c, infoPort=40855, ipcPort=36581, storageInfo=lv=-56;cid=testClusterID;nsid=911232124;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 705492 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 705688 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59784
   [junit4]   2> 705900 INFO  (IPC Server handler 5 on 58696) [    ] BlockStateChange BLOCK* processReport: from storage DS-aa26c54e-7058-44a3-bfd9-8d922be4a1d2 node DatanodeRegistration(127.0.0.1, datanodeUuid=6300e33a-b4ca-4d3b-8167-df5a8e970f8f, infoPort=59784, ipcPort=37865, storageInfo=lv=-56;cid=testClusterID;nsid=911232124;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 705900 INFO  (IPC Server handler 5 on 58696) [    ] BlockStateChange BLOCK* processReport: from storage DS-b9f7b83a-4810-4bba-bd5c-bfba988f8972 node DatanodeRegistration(127.0.0.1, datanodeUuid=6300e33a-b4ca-4d3b-8167-df5a8e970f8f, infoPort=59784, ipcPort=37865, storageInfo=lv=-56;cid=testClusterID;nsid=911232124;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 706082 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 706084 INFO  (Thread-3618) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 706084 INFO  (Thread-3618) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 706184 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkTestServer start zk server on port:34302
   [junit4]   2> 706184 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 706184 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 706192 INFO  (zkCallback-1573-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6911a79 name:ZooKeeperConnection Watcher:127.0.0.1:34302 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706192 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 706192 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 706192 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 706200 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 706210 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 706210 INFO  (zkCallback-1574-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@56846c0b name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706210 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 706211 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 706211 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 706214 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 706216 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 706217 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 706219 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 706219 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 706221 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 706221 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 706223 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 706223 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 706224 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 706224 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 706226 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 706226 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 706227 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 706227 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 706229 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 706229 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 706230 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 706230 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 706232 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 706232 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 706234 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 706234 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 706235 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 706236 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 706608 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores/collection1
   [junit4]   2> 706610 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 706618 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3a93d624{/,null,AVAILABLE}
   [junit4]   2> 706618 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6d7d44cb{HTTP/1.1}{127.0.0.1:46663}
   [junit4]   2> 706618 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.Server Started @709456ms
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores, solr.data.dir=hdfs://localhost:58696/hdfs__localhost_58696__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001_tempDir-002_control_data, hostPort=46663, hostContext=/}
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001'
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 706619 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 706620 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 706628 INFO  (zkCallback-1575-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7dd6c24 name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706629 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 706629 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 706632 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15185de05b70002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 706632 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 706632 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/solr.xml
   [junit4]   2> 706653 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores
   [junit4]   2> 706653 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer New CoreContainer 134746427
   [junit4]   2> 706653 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001]
   [junit4]   2> 706653 WARN  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/lib
   [junit4]   2> 706654 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 706655 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 706656 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 706656 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 706656 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34302/solr
   [junit4]   2> 706656 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 706656 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 706658 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 706658 INFO  (zkCallback-1577-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@37708dc5 name:ZooKeeperConnection Watcher:127.0.0.1:34302 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706660 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 706660 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 706664 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 706680 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50d1b0da name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 706680 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 706682 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 706684 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 706687 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 706689 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 706691 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 706696 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 706701 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 706704 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 706706 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 706711 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46663_
   [junit4]   2> 706711 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46663_
   [junit4]   2> 706714 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 706718 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 706720 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 706720 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95004255776604164-127.0.0.1:46663_-n_0000000000
   [junit4]   2> 706721 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46663_
   [junit4]   2> 706721 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 706722 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer Overseer (id=95004255776604164-127.0.0.1:46663_-n_0000000000) starting
   [junit4]   2> 706727 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 706740 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 706742 INFO  (OverseerCollectionConfigSetProcessor-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 706744 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 706744 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 706779 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 706779 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 706781 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores
   [junit4]   2> 706782 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {shard=, configSetProperties=configsetprops.json, collection=control_collection, loadOnStartup=true, dataDir=data/, config=solrconfig.xml, transient=false, schema=schema.xml, coreNodeName=, name=collection1}
   [junit4]   2> 706782 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores/collection1
   [junit4]   2> 706782 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 706789 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 706789 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:46663_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 706789 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 706789 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 706793 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 706794 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:46663_",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46663"} current state version: 0
   [junit4]   2> 706794 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:46663_",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46663"}
   [junit4]   2> 706795 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 706795 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 706798 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 707794 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 707795 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 707796 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 707796 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 707797 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 707797 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores/collection1'
   [junit4]   2> 707797 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 707797 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 707807 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 707817 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 707830 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 707863 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 707878 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 708284 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 708296 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 708298 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 708322 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 708327 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 708330 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 708331 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 708331 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 708331 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 708332 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 708332 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 708333 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 708333 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 708333 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 708344 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:58696/solr_hdfs_home
   [junit4]   2> 708344 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 708344 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/control-001/cores/collection1], dataDir=[null]
   [junit4]   2> 708345 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@64c3d9c0
   [junit4]   2> 708345 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 708434 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 708449 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 708477 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 708477 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 708505 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 708506 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 708506 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 708559 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 708559 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 708559 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=1.5556640625, floorSegmentMB=2.083984375, forceMergeDeletesPctAllowed=2.632039295550377, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 708615 INFO  (IPC Server handler 5 on 58696) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36944 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-13b02519-4d94-4f73-a6d3-618de83dacac:NORMAL:127.0.0.1:42534|RBW], ReplicaUnderConstruction[[DISK]DS-aa26c54e-7058-44a3-bfd9-8d922be4a1d2:NORMAL:127.0.0.1:36944|RBW]]} size 0
   [junit4]   2> 708619 INFO  (IPC Server handler 6 on 58696) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42534 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-aa26c54e-7058-44a3-bfd9-8d922be4a1d2:NORMAL:127.0.0.1:36944|RBW], ReplicaUnderConstruction[[DISK]DS-434609a3-3092-4032-8859-72cd30a2283f:NORMAL:127.0.0.1:42534|FINALIZED]]} size 0
   [junit4]   2> 708632 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71de19d8); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 708632 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 708635 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 708639 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 708640 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 708640 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 708640 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 708641 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 708641 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 708641 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 708641 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 708641 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 708641 WARN  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 708644 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 708645 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 708655 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 708663 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 708670 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/threads,/admin/system,/admin/properties,/get,/update/csv,/admin/logging,/admin/file,/update/json,/admin/plugins,standard,/update/json/docs,/admin/segments,/admin/ping,/update,/schema,/admin/mbeans,/replication,/config,/admin/luke,/dump
   [junit4]   2> 708672 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 708678 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 708679 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 708702 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 708702 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 708704 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=32.3037109375, floorSegmentMB=0.3125, forceMergeDeletesPctAllowed=12.00496103389468, segmentsPerTier=32.0, maxCFSSegmentSizeMB=1.078125, noCFSRatio=1.0
   [junit4]   2> 708810 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:58696/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71de19d8); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 708810 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 708811 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@f541914[collection1] main
   [junit4]   2> 708811 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 708813 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 708813 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 708813 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 708813 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 708813 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 708814 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 708814 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 708814 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 708817 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 708820 INFO  (searcherExecutor-4046-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@f541914[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version buckets from index
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1520068095287230464
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 1520068095287230464
   [junit4]   2> 708822 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 708823 INFO  (coreLoadExecutor-4045-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 708824 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:46663 collection:control_collection shard:shard1
   [junit4]   2> 708828 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 708833 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95004255776604164-core_node1-n_0000000000
   [junit4]   2> 708834 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 708836 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 708836 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 708836 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46663/collection1/
   [junit4]   2> 708836 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 708836 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:46663/collection1/ has no replicas
   [junit4]   2> 708837 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 708838 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 708839 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 708844 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 708846 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46663/collection1/ shard1
   [junit4]   2> 708848 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46663",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 708951 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 708998 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:46663/collection1/ and leader is http://127.0.0.1:46663/collection1/
   [junit4]   2> 708999 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46663
   [junit4]   2> 708999 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 708999 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 708999 INFO  (coreZkRegister-4039-thread-1-processing-n:127.0.0.1:46663_ s:shard1 x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:46663_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 709004 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:46663_",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46663"} current state version: 3
   [junit4]   2> 709005 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:46663_",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:46663"}
   [junit4]   2> 709111 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 709292 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 709293 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 709294 INFO  (zkCallback-1580-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2f0fa4ae name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 709294 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 709294 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 709294 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 709298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 709298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 709298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 709298 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 709300 INFO  (zkCallback-1581-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@347edc46 name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 709300 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 709301 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 709306 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 4
   [junit4]   2> 709306 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 709422 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 709652 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/cores/collection1
   [junit4]   2> 709652 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001
   [junit4]   2> 709653 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 709668 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@327a9510{/,null,AVAILABLE}
   [junit4]   2> 709669 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6d2fc2e1{HTTP/1.1}{127.0.0.1:60403}
   [junit4]   2> 709669 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.e.j.s.Server Started @712507ms
   [junit4]   2> 709669 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, solr.data.dir=hdfs://localhost:58696/hdfs__localhost_58696__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001_tempDir-002_jetty1, hostPort=60403, hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/cores}
   [junit4]   2> 709669 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 709669 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001'
   [junit4]   2> 709670 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 709670 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 709670 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 709676 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 709676 INFO  (zkCallback-1582-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5c94f01e name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 709676 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 709677 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 709681 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 709681 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/solr.xml
   [junit4]   2> 709696 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/cores
   [junit4]   2> 709696 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer New CoreContainer 2146791045
   [junit4]   2> 709696 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001]
   [junit4]   2> 709696 WARN  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/lib
   [junit4]   2> 709698 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34302/solr
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 709700 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 709703 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 709707 INFO  (zkCallback-1584-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c11ac42 name:ZooKeeperConnection Watcher:127.0.0.1:34302 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 709708 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 709708 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 709711 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x15185de05b70008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 709716 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 709718 INFO  (zkCallback-1585-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7dceea47 name:ZooKeeperConnection Watcher:127.0.0.1:34302/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 709718 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 709728 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 710741 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60403_
   [junit4]   2> 710742 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:60403_
   [junit4]   2> 710743 INFO  (zkCallback-1580-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 710743 INFO  (zkCallback-1585-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 710743 INFO  (zkCallback-1578-thread-1-processing-n:127.0.0.1:46663_) [n:127.0.0.1:46663_    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 710746 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 710747 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95004255776604169-127.0.0.1:60403_-n_0000000001
   [junit4]   2> 710749 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95004255776604164-127.0.0.1:46663_-n_0000000000 to know if I could be the leader
   [junit4]   2> 710765 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 710765 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 710766 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/cores
   [junit4]   2> 710766 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {schema=schema.xml, loadOnStartup=true, coreNodeName=, shard=, collection=collection1, name=collection1, config=solrconfig.xml, transient=false, configSetProperties=configsetprops.json, dataDir=data/}
   [junit4]   2> 710767 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001/shard-1-001/cores/collection1
   [junit4]   2> 710767 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 710769 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 710769 INFO  (TEST-HdfsBasicDistributedZk2Test.test-seed#[BF6DB73668EA9BBA]) [n:127.0.0.1:60403_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 710771 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 710771 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 710773 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 710773 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:60403_",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:60403"} current state version: 4
   [junit4]   2> 710773 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch collection1
   [junit4]   2> 710774 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "node_name":"127.0.0.1:60403_",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:60403"}
   [junit4]   2> 710774 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=2
   [junit4]   2> 710774 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 710774 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for collection1 ver 0 
   [junit4]   2> 710774 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 710876 INFO  (OverseerStateUpdate-95004255776604164-127.0.0.1:46663_-n_0000000000) [n:127.0.0.1:46663_    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 710877 INFO  (zkCallback-1585-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 710877 INFO  (zkCallback-1585-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_    ] o.a.s.c.c.ZkStateReader Updating data for collection1 from 0 to 1 
   [junit4]   2> 711774 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 711774 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 711775 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 711775 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/collection1
   [junit4]   2> 711776 INFO  (coreLoadExecutor-4056-thread-1-processing-n:127.0.0.1:60403_) [n:127.0.0.1:60403_ c:collection1   x:coll

[...truncated too long message...]

terceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsBasicDistributedZk2Test.teardownClass(HdfsBasicDistributedZk2Test.java:47)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 809787 INFO  (SUITE-HdfsBasicDistributedZk2Test-seed#[BF6DB73668EA9BBA]-worker) [n:127.0.0.1:59573_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsBasicDistributedZk2Test_BF6DB73668EA9BBA-001
   [junit4]   2> Dec 09, 2015 8:33:22 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {foo_b=FST50, n_l1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_d1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_tdt1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_i_dv=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_ti1=PostingsFormat(name=Memory doPackFST= false), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=Lucene50(blocksize=128), id=PostingsFormat(name=Memory doPackFST= false), foo_f=FST50, n_tl1=Lucene50(blocksize=128), rnd_b=PostingsFormat(name=Memory doPackFST= false), range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), foo_d=PostingsFormat(name=Memory doPackFST= false), n_td1=Lucene50(blocksize=128), oddField_s=FST50, timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text=FST50, SubjectTerms_mfacet=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_tf1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_f1=Lucene50(blocksize=128), n_dt1=Lucene50(blocksize=128), range_facet_l_dv=PostingsFormat(name=Memory doPackFST= false), other_tl1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Memory)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=it_CH, timezone=Asia/Riyadh
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=5,free=268238792,total=466616320
   [junit4]   2> NOTE: All tests run in this JVM: [HighlighterMaxOffsetTest, TestSolrCoreProperties, TestSortByMinMaxFunction, SpellingQueryConverterTest, TestSurroundQueryParser, AnalyticsMergeStrategyTest, CollectionsAPIDistributedZkTest, TestStressVersions, SimplePostToolTest, ChaosMonkeySafeLeaderTest, TestJsonFacets, DistanceFunctionTest, TestRandomRequestDistribution, TestCloudPivotFacet, MoreLikeThisHandlerTest, TestRemoveLastDynamicCopyField, UpdateRequestProcessorFactoryTest, TestWordDelimiterFilterFactory, TestSolrConfigHandlerCloud, TestLRUStatsCache, TestSolrQueryParserDefaultOperatorResource, TestRawTransformer, TestHashQParserPlugin, HdfsThreadLeakTest, TestTrie, TestReplicationHandlerBackup, TestDistributedGrouping, TestLeaderElectionZkExpiry, TestDFRSimilarityFactory, BasicFunctionalityTest, TestMissingGroups, IndexBasedSpellCheckerTest, HdfsBasicDistributedZk2Test]
   [junit4] Completed [102/553 (1!)] on J1 in 115.89s, 1 test, 1 error <<< FAILURES!

[...truncated 1444 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1456: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1010: There were test failures: 553 suites (7 ignored), 2188 tests, 1 error, 33 ignored (18 assumptions) [seed: BF6DB73668EA9BBA]

Total time: 203 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1040 - Still Failing

[Mikhail Khludnev <mk...@griddynamics.com>]

Got. I just didn't have jdk7 after last laptop crash. 5x build on jdk8
caught syntax errors, that's why I relaxed after fixing compile errors and
run tests. I didn't catch rt.jar difference. Now I'm able to test on jdk7.
Awaiting for CI.

On Tue, Dec 8, 2015 at 5:25 PM, david.w.smiley@gmail.com <
david.w.smiley@gmail.com> wrote:

> Mikhail,
> In the future, particularly when committing code that uses Java 8
> features, be sure to compile & run tests even on the 5x branch after
> back-porting but before committing.  From time to time this fails for
> stuff, like easy-to-forget cases of not using ‘final’.
> ~ David
>
> On Tue, Dec 8, 2015 at 2:21 AM Mikhail Khludnev <
> mkhludnev@griddynamics.com> wrote:
>
>> It's mine. Looking into.
>>
>> On Tue, Dec 8, 2015 at 9:35 AM, Apache Jenkins Server <
>> jenkins@builds.apache.org> wrote:
>>
>>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1040/
>>>
>>> All tests passed
>>>
>>> Build Log:
>>> [...truncated 7450 lines...]
>>>     [javac] Compiling 6 source files to
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/join/classes/test
>>>     [javac]
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:954:
>>> error: cannot find symbol
>>>     [javac]         assert nextInt ==
>>> Integer.parseUnsignedInt(uniqueRandomValue,16);
>>>     [javac]                                  ^
>>>     [javac]   symbol:   method parseUnsignedInt(String,int)
>>>     [javac]   location: class Integer
>>>     [javac]
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:1206:
>>> error: cannot find symbol
>>>     [javac]     final int linkInt =
>>> Integer.parseUnsignedInt(linkValue,16);
>>>     [javac]                                ^
>>>     [javac]   symbol:   method parseUnsignedInt(String,int)
>>>     [javac]   location: class Integer
>>>     [javac] Note:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
>>> uses or overrides a deprecated API.
>>>     [javac] Note: Recompile with -Xlint:deprecation for details.
>>>     [javac] 2 errors
>>>
>>> BUILD FAILED
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2252:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:55:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:818:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:832:
>>> The following error occurred while executing this line:
>>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1968:
>>> Compile failed; see the compiler error output for details.
>>>
>>> Total time: 106 minutes 8 seconds
>>> Build step 'Invoke Ant' marked build as failure
>>> Archiving artifacts
>>> Recording test results
>>> Email was triggered for: Failure - Any
>>> Sending email for trigger: Failure - Any
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> Principal Engineer,
>> Grid Dynamics
>>
>> <http://www.griddynamics.com>
>> <mk...@griddynamics.com>
>>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>

Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1040 - Still Failing

["david.w.smiley@gmail.com" <da...@gmail.com>]

Mikhail,
In the future, particularly when committing code that uses Java 8 features,
be sure to compile & run tests even on the 5x branch after back-porting but
before committing.  From time to time this fails for stuff, like
easy-to-forget cases of not using ‘final’.
~ David

On Tue, Dec 8, 2015 at 2:21 AM Mikhail Khludnev <mk...@griddynamics.com>
wrote:

> It's mine. Looking into.
>
> On Tue, Dec 8, 2015 at 9:35 AM, Apache Jenkins Server <
> jenkins@builds.apache.org> wrote:
>
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1040/
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 7450 lines...]
>>     [javac] Compiling 6 source files to
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/join/classes/test
>>     [javac]
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:954:
>> error: cannot find symbol
>>     [javac]         assert nextInt ==
>> Integer.parseUnsignedInt(uniqueRandomValue,16);
>>     [javac]                                  ^
>>     [javac]   symbol:   method parseUnsignedInt(String,int)
>>     [javac]   location: class Integer
>>     [javac]
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:1206:
>> error: cannot find symbol
>>     [javac]     final int linkInt =
>> Integer.parseUnsignedInt(linkValue,16);
>>     [javac]                                ^
>>     [javac]   symbol:   method parseUnsignedInt(String,int)
>>     [javac]   location: class Integer
>>     [javac] Note:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
>> uses or overrides a deprecated API.
>>     [javac] Note: Recompile with -Xlint:deprecation for details.
>>     [javac] 2 errors
>>
>> BUILD FAILED
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2252:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:55:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:818:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:832:
>> The following error occurred while executing this line:
>> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1968:
>> Compile failed; see the compiler error output for details.
>>
>> Total time: 106 minutes 8 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <mk...@griddynamics.com>
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1040 - Still Failing

[Mikhail Khludnev <mk...@griddynamics.com>]

It's mine. Looking into.

On Tue, Dec 8, 2015 at 9:35 AM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1040/
>
> All tests passed
>
> Build Log:
> [...truncated 7450 lines...]
>     [javac] Compiling 6 source files to
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/join/classes/test
>     [javac]
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:954:
> error: cannot find symbol
>     [javac]         assert nextInt ==
> Integer.parseUnsignedInt(uniqueRandomValue,16);
>     [javac]                                  ^
>     [javac]   symbol:   method parseUnsignedInt(String,int)
>     [javac]   location: class Integer
>     [javac]
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:1206:
> error: cannot find symbol
>     [javac]     final int linkInt = Integer.parseUnsignedInt(linkValue,16);
>     [javac]                                ^
>     [javac]   symbol:   method parseUnsignedInt(String,int)
>     [javac]   location: class Integer
>     [javac] Note:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java
> uses or overrides a deprecated API.
>     [javac] Note: Recompile with -Xlint:deprecation for details.
>     [javac] 2 errors
>
> BUILD FAILED
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2252:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:55:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:818:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:832:
> The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1968:
> Compile failed; see the compiler error output for details.
>
> Total time: 106 minutes 8 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1040 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1040/

All tests passed

Build Log:
[...truncated 7450 lines...]
    [javac] Compiling 6 source files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/join/classes/test
    [javac] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:954: error: cannot find symbol
    [javac]         assert nextInt == Integer.parseUnsignedInt(uniqueRandomValue,16);
    [javac]                                  ^
    [javac]   symbol:   method parseUnsignedInt(String,int)
    [javac]   location: class Integer
    [javac] /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestJoinUtil.java:1206: error: cannot find symbol
    [javac]     final int linkInt = Integer.parseUnsignedInt(linkValue,16);
    [javac]                                ^
    [javac]   symbol:   method parseUnsignedInt(String,int)
    [javac]   location: class Integer
    [javac] Note: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/join/src/test/org/apache/lucene/search/join/TestBlockJoin.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] 2 errors

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:801: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:738: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2252: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:55: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:818: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:832: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1968: Compile failed; see the compiler error output for details.

Total time: 106 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1039 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1039/

3 tests failed.
FAILED:  org.apache.solr.cloud.TestAuthenticationFramework.testCollectionCreateSearchDelete

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at __randomizedtesting.SeedInfo.seed([6B4A28DD8930C974:C8B086780ED823D1]:0)
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:463)
	at sun.nio.ch.Net.bind(Net.java:455)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
	at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.server.Server.doStart(Server.java:366)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:407)
	at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:357)
	at org.apache.solr.cloud.TestMiniSolrCloudCluster.testCollectionCreateSearchDelete(TestMiniSolrCloudCluster.java:190)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestAuthenticationFramework

Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.TestAuthenticationFramework:     1) Thread[id=50657, name=qtp1598845069-50657, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=50661, name=Scheduler-304156586, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=50656, name=qtp1598845069-50656, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=50655, name=qtp1598845069-50655, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=50658, name=qtp1598845069-50658, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=50654, name=qtp1598845069-50654, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=50660, name=qtp1598845069-50660, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=50653, name=qtp1598845069-50653, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=50659, name=qtp1598845069-50659, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.TestAuthenticationFramework: 
   1) Thread[id=50657, name=qtp1598845069-50657, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=50661, name=Scheduler-304156586, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=50656, name=qtp1598845069-50656, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=50655, name=qtp1598845069-50655, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=50658, name=qtp1598845069-50658, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=50654, name=qtp1598845069-50654, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=50660, name=qtp1598845069-50660, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=50653, name=qtp1598845069-50653, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=50659, name=qtp1598845069-50659, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6B4A28DD8930C974]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestAuthenticationFramework

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=50661, name=Scheduler-304156586, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=50661, name=Scheduler-304156586, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([6B4A28DD8930C974]:0)




Build Log:
[...truncated 10595 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestAuthenticationFramework
   [junit4]   2> 855113 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 855114 INFO  (Thread-48265) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 855114 INFO  (Thread-48265) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 855214 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.ZkTestServer start zk server on port:51098
   [junit4]   2> 855214 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855214 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855223 INFO  (zkCallback-386-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c315d15 name:ZooKeeperConnection Watcher:127.0.0.1:51098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855223 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855223 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855224 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 855230 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 855230 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 855231 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 855234 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@689a6a7e{/solr,null,AVAILABLE}
   [junit4]   2> 855234 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@37db9ed2{/solr,null,AVAILABLE}
   [junit4]   2> 855234 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@339809a4{/solr,null,AVAILABLE}
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@6d926c48{HTTP/1.1}{127.0.0.1:55066}
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.Server Started @858130ms
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=55066, hostContext=/solr}
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1'
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 855238 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 855239 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855239 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855239 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@65f9f092{HTTP/1.1}{127.0.0.1:48544}
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.Server Started @858132ms
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=48544, hostContext=/solr}
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@c999844{HTTP/1.1}{127.0.0.1:55562}
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.Server Started @858132ms
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=55562, hostContext=/solr}
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 855240 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2'
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3'
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855241 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855242 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855244 INFO  (zkCallback-387-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7f2809d6 name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855244 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855244 INFO  (zkCallback-388-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@579a9cba name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855244 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855244 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855244 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855245 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 855250 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 855250 INFO  (zkCallback-389-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@44c4ad4c name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855250 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855250 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855252 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 855267 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2/.
   [junit4]   2> 855267 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 399278526
   [junit4]   2> 855267 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2]
   [junit4]   2> 855267 WARN  (jetty-launcher-385-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2/lib
   [junit4]   2> 855268 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 855268 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1/.
   [junit4]   2> 855268 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 547657681
   [junit4]   2> 855269 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1]
   [junit4]   2> 855269 WARN  (jetty-launcher-385-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1/lib
   [junit4]   2> 855269 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 855270 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 855271 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51098/solr
   [junit4]   2> 855271 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51098/solr
   [junit4]   2> 855271 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 855272 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 855272 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855272 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855273 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855273 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3/.
   [junit4]   2> 855273 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 54608992
   [junit4]   2> 855273 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3]
   [junit4]   2> 855273 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855273 WARN  (jetty-launcher-385-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3/lib
   [junit4]   2> 855274 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 855274 INFO  (zkCallback-393-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40e303e4 name:ZooKeeperConnection Watcher:127.0.0.1:51098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855274 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 855274 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51098/solr
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 855275 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855275 INFO  (zkCallback-392-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ea0ae38 name:ZooKeeperConnection Watcher:127.0.0.1:51098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855280 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855280 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855280 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855281 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855282 INFO  (zkCallback-395-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43833910 name:ZooKeeperConnection Watcher:127.0.0.1:51098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855282 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855283 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855291 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855291 INFO  (zkCallback-396-thread-1-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@703a309f name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855291 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855291 INFO  (zkCallback-397-thread-1-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7073b9fc name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855291 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855292 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855294 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 855294 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 855294 INFO  (zkCallback-398-thread-1-processing-n:127.0.0.1:55562_solr) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@685b0835 name:ZooKeeperConnection Watcher:127.0.0.1:51098/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855294 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855300 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 855300 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 855301 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 855307 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 855308 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 855308 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 855311 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 855311 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 855316 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 855316 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 855320 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 855320 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 855320 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 855325 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 855325 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 855325 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 855331 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 855331 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 855332 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 855334 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 855335 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 855335 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 855337 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 855337 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 855337 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 855339 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48544_solr
   [junit4]   2> 855339 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55562_solr
   [junit4]   2> 855340 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55066_solr
   [junit4]   2> 855340 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:48544_solr
   [junit4]   2> 855341 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55562_solr
   [junit4]   2> 855341 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:55066_solr
   [junit4]   2> 855343 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 855344 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 855346 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 855346 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 855346 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 855348 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 855348 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 855348 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 855350 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94992915367526407-127.0.0.1:55066_solr-n_0000000000
   [junit4]   2> 855350 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94992915367526409-127.0.0.1:55562_solr-n_0000000001
   [junit4]   2> 855350 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94992915367526408-127.0.0.1:48544_solr-n_0000000002
   [junit4]   2> 855350 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55066_solr
   [junit4]   2> 855350 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 855351 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94992915367526407-127.0.0.1:55066_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 855351 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 855351 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94992915367526409-127.0.0.1:55562_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 855351 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 855353 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.Overseer Overseer (id=94992915367526407-127.0.0.1:55066_solr-n_0000000000) starting
   [junit4]   2> 855364 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 855364 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 855365 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.CoreContainer Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 855365 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@28ba9edb
   [junit4]   2> 855365 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@28ba9edb
   [junit4]   2> 855366 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node2/.
   [junit4]   2> 855366 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 855374 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 855374 INFO  (jetty-launcher-385-thread-2) [n:127.0.0.1:48544_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 855376 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 855381 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 855381 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 855382 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.CoreContainer Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 855382 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@404c9413
   [junit4]   2> 855382 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@404c9413
   [junit4]   2> 855383 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node3/.
   [junit4]   2> 855384 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 855389 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 855389 INFO  (jetty-launcher-385-thread-3) [n:127.0.0.1:55562_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 855394 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 855394 INFO  (OverseerCollectionConfigSetProcessor-94992915367526407-127.0.0.1:55066_solr-n_0000000000) [n:127.0.0.1:55066_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 855394 INFO  (OverseerStateUpdate-94992915367526407-127.0.0.1:55066_solr-n_0000000000) [n:127.0.0.1:55066_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 855395 INFO  (OverseerStateUpdate-94992915367526407-127.0.0.1:55066_solr-n_0000000000) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 855404 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 855430 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 855430 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin
   [junit4]   2> 855430 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.CoreContainer Reconfiguring the shard handler factory and update shard handler.
   [junit4]   2> 855431 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.h.c.HttpShardHandlerFactory Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@40284289
   [junit4]   2> 855431 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.u.UpdateShardHandler Reconfiguring the default client with: org.apache.solr.cloud.TestAuthenticationFramework$MockAuthenticationPlugin$MockClientConfigurer@40284289
   [junit4]   2> 855432 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-001/node1/.
   [junit4]   2> 855433 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 855433 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2
   [junit4]   2> 855434 INFO  (jetty-launcher-385-thread-1) [n:127.0.0.1:55066_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 855434 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 855434 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 855436 INFO  (zkCallback-399-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c79c234 name:ZooKeeperConnection Watcher:127.0.0.1:51098 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 855436 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 855437 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 855441 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@c999844{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 855441 INFO  (jetty-launcher-385-thread-3) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=54608992
   [junit4]   2> 855443 INFO  (zkCallback-396-thread-1-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 855444 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@6d926c48{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 855448 ERROR (jetty-launcher-385-thread-3-EventThread) [n:127.0.0.1:55562_solr    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@d1129e1 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@61d4797a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:215)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:261)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 855450 INFO  (jetty-launcher-385-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@689a6a7e{/solr,null,UNAVAILABLE}
   [junit4]   2> 855449 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=547657681
   [junit4]   2> 855448 INFO  (zkCallback-396-thread-2-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 855451 INFO  (zkCallback-397-thread-1-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 855457 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@65f9f092{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 855460 INFO  (jetty-launcher-385-thread-2) [    ] o.a.s.c.Overseer Overseer (id=94992915367526407-127.0.0.1:55066_solr-n_0000000000) closing
   [junit4]   2> 855460 INFO  (OverseerStateUpdate-94992915367526407-127.0.0.1:55066_solr-n_0000000000) [n:127.0.0.1:55066_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:55066_solr
   [junit4]   2> 855460 INFO  (jetty-launcher-385-thread-1) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=399278526
   [junit4]   2> 855464 INFO  (zkCallback-397-thread-1-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 855464 WARN  (zkCallback-397-thread-2-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.LeaderElector Failed setting watch
   [junit4]   2> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer_elect/election/94992915367526407-127.0.0.1:55066_solr-n_0000000000
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:182)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:59)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:389)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 855464 INFO  (zkCallback-396-thread-1-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 855465 INFO  (zkCallback-396-thread-2-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 855465 WARN  (zkCallback-396-thread-2-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 856964 WARN  (zkCallback-397-thread-1-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 856965 WARN  (zkCallback-396-thread-1-processing-n:127.0.0.1:55066_solr) [n:127.0.0.1:55066_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 856966 WARN  (zkCallback-397-thread-2-processing-n:127.0.0.1:48544_solr) [n:127.0.0.1:48544_solr    ] o.a.s.c.LeaderElector 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:190)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:59)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:389)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 856966 INFO  (jetty-launcher-385-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@37db9ed2{/solr,null,UNAVAILABLE}
   [junit4]   2> 856968 INFO  (jetty-launcher-385-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@339809a4{/solr,null,UNAVAILABLE}
   [junit4]   2> 856972 INFO  (TEST-TestAuthenticationFramework.testErrorsInShutdown-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51098 51098
   [junit4]   2> 857194 INFO  (Thread-48265) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51098 51098
   [junit4]   2> 857195 WARN  (Thread-48265) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 857202 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 857202 INFO  (Thread-48302) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 857202 INFO  (Thread-48302) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 857302 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.ZkTestServer start zk server on port:45652
   [junit4]   2> 857302 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857303 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857310 INFO  (zkCallback-402-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1006d0ee name:ZooKeeperConnection Watcher:127.0.0.1:45652 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857311 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857311 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857311 INFO  (TEST-TestAuthenticationFramework.testCollectionCreateSearchDelete-seed#[6B4A28DD8930C974]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 857320 INFO  (jetty-launcher-401-thread-1) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 857320 INFO  (jetty-launcher-401-thread-5) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 857320 INFO  (jetty-launcher-401-thread-2) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 857322 INFO  (jetty-launcher-401-thread-4) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 857322 INFO  (jetty-launcher-401-thread-3) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 857324 INFO  (jetty-launcher-401-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@50fcf69a{/solr,null,AVAILABLE}
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@4a64f75f{HTTP/1.1}{127.0.0.1:57569}
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.e.j.s.Server Started @860217ms
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57569}
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node1'
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857325 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6ea123a9{/solr,null,AVAILABLE}
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.e.j.s.ServerConnector Started ServerConnector@159a28e7{HTTP/1.1}{127.0.0.1:51048}
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.e.j.s.Server Started @860218ms
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=51048, hostContext=/solr}
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 857326 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node5'
   [junit4]   2> 857327 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857327 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857327 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857327 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857328 INFO  (jetty-launcher-401-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8d2b986{/solr,null,AVAILABLE}
   [junit4]   2> 857329 INFO  (jetty-launcher-401-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2875274a{/solr,null,AVAILABLE}
   [junit4]   2> 857329 INFO  (jetty-launcher-401-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@3ef03973{HTTP/1.1}{127.0.0.1:49406}
   [junit4]   2> 857329 INFO  (jetty-launcher-401-thread-2) [    ] o.e.j.s.Server Started @860221ms
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=49406}
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node2'
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857330 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857331 INFO  (zkCallback-404-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14f71064 name:ZooKeeperConnection Watcher:127.0.0.1:45652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857331 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857331 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857331 INFO  (zkCallback-403-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d3420 name:ZooKeeperConnection Watcher:127.0.0.1:45652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857331 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857332 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 857332 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857332 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 857332 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@3a3a0c92{HTTP/1.1}{127.0.0.1:57931}
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.e.j.s.Server Started @860225ms
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57931}
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node3'
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857333 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857334 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857336 INFO  (jetty-launcher-401-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@54d573b6{/solr,null,AVAILABLE}
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@99fdd3d{HTTP/1.1}{127.0.0.1:44879}
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.e.j.s.Server Started @860229ms
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44879}
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@3827de24
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node4'
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857337 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857338 INFO  (zkCallback-405-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@177e48ed name:ZooKeeperConnection Watcher:127.0.0.1:45652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857338 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857338 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857339 INFO  (zkCallback-406-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5abcee89 name:ZooKeeperConnection Watcher:127.0.0.1:45652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857339 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857339 INFO  (jetty-launcher-401-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 857339 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857339 INFO  (jetty-launcher-401-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 857344 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857345 INFO  (zkCallback-407-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1beffb48 name:ZooKeeperConnection Watcher:127.0.0.1:45652/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 857346 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 857346 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 857346 INFO  (jetty-launcher-401-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 857351 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node1/.
   [junit4]   2> 857352 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node5/.
   [junit4]   2> 857352 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.CoreContainer New CoreContainer 844108746
   [junit4]   2> 857352 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node5]
   [junit4]   2> 857352 WARN  (jetty-launcher-401-thread-5) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node5/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node5/lib
   [junit4]   2> 857353 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 857354 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45652/solr
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 857355 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 626263853
   [junit4]   2> 857356 INFO  (jetty-launcher-401-thread-5) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 857360 INFO  (jetty-launcher-401-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node1]
   [junit4]   2> 857361 WARN  (jetty-launcher-401-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.TestAuthenticationFramework_6B4A28DD8930C974-001/tempDir-002/node1/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Nigh

[...truncated too long message...]

0(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=50658, name=qtp1598845069-50658, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=50654, name=qtp1598845069-50654, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    7) Thread[id=50660, name=qtp1598845069-50660, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    8) Thread[id=50653, name=qtp1598845069-50653, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    9) Thread[id=50659, name=qtp1598845069-50659, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   10) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6B4A28DD8930C974]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=50661, name=Scheduler-304156586, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=50662, name=org.eclipse.jetty.server.session.HashSessionManager@25ab2976Timer, state=TIMED_WAITING, group=TGRP-TestAuthenticationFramework]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4]    >         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6B4A28DD8930C974]:0)
   [junit4] Completed [131/553] on J2 in 123.11s, 7 tests, 3 errors <<< FAILURES!

[...truncated 1295 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:799: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:736: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 553 suites (7 ignored), 2187 tests, 2 suite-level errors, 1 error, 1252 ignored (15 assumptions) [seed: 6B4A28DD8930C974]

Total time: 231 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1038 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1038/

2 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1449417291014,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]> but was:<[{indexVersion=1449417291014,generation=3,filelist=[_6.cfe, _6.cfs, _6.si, segments_3]}, {indexVersion=1449417291014,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1449417291014,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]> but was:<[{indexVersion=1449417291014,generation=3,filelist=[_6.cfe, _6.cfs, _6.si, segments_3]}, {indexVersion=1449417291014,generation=2,filelist=[_0.fdt, _0.fdx, _0.fnm, _0.nvd, _0.nvm, _0.si, _0_Lucene50_0.doc, _0_Lucene50_0.tim, _0_Lucene50_0.tip, _1.fdt, _1.fdx, _1.fnm, _1.nvd, _1.nvm, _1.si, _1_Lucene50_0.doc, _1_Lucene50_0.tim, _1_Lucene50_0.tip, _2.fdt, _2.fdx, _2.fnm, _2.nvd, _2.nvm, _2.si, _2_Lucene50_0.doc, _2_Lucene50_0.tim, _2_Lucene50_0.tip, _3.fdt, _3.fdx, _3.fnm, _3.nvd, _3.nvm, _3.si, _3_Lucene50_0.doc, _3_Lucene50_0.tim, _3_Lucene50_0.tip, _4.fdt, _4.fdx, _4.fnm, _4.nvd, _4.nvm, _4.si, _4_Lucene50_0.doc, _4_Lucene50_0.tim, _4_Lucene50_0.tip, _5.fdt, _5.fdx, _5.fnm, _5.nvd, _5.nvm, _5.si, _5_Lucene50_0.doc, _5_Lucene50_0.tim, _5_Lucene50_0.tip, segments_2]}]>
	at __randomizedtesting.SeedInfo.seed([DA5873B6F3FA9AF8:FF8F688683B294FB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1140)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Mon Dec 07 02:56:23 SBT 2015

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Mon Dec 07 02:56:23 SBT 2015
	at __randomizedtesting.SeedInfo.seed([DA5873B6F3FA9AF8:1F37370F6D2F34B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1419)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:771)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10731 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/init-core-data-001
   [junit4]   2> 1225543 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.SolrTestCaseJ4 ###Starting doTestRepeater
   [junit4]   2> 1225544 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1
   [junit4]   2> 1225580 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1225637 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@108ae9d5{/solr,null,AVAILABLE}
   [junit4]   2> 1225654 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.ServerConnector Started ServerConnector@55fb709c{HTTP/1.1}{127.0.0.1:34195}
   [junit4]   2> 1225654 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server Started @1228780ms
   [junit4]   2> 1225654 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=34195, hostContext=/solr, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1/data}
   [junit4]   2> 1225654 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4f6de641
   [junit4]   2> 1225654 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001'
   [junit4]   2> 1225655 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1225655 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1225655 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/solr.xml
   [junit4]   2> 1225699 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/.
   [junit4]   2> 1225699 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer New CoreContainer 1822242929
   [junit4]   2> 1225699 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001]
   [junit4]   2> 1225700 WARN  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/lib
   [junit4]   2> 1225701 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1225703 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1225704 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1225704 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1225705 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1225705 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1225707 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/.
   [junit4]   2> 1225715 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, transient=false, loadOnStartup=true, config=solrconfig.xml, dataDir=data/, configSetProperties=configsetprops.json, schema=schema.xml}
   [junit4]   2> 1225716 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1
   [junit4]   2> 1225716 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1225727 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1'
   [junit4]   2> 1225738 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1225744 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1225744 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1225744 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1225753 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1225766 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1225790 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1225795 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1225814 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1225817 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1'
   [junit4]   2> 1225817 INFO  (coreLoadExecutor-4439-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1
   [junit4]   2> 1225817 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1225818 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1], dataDir=[null]
   [junit4]   2> 1225818 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1225818 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1/data
   [junit4]   2> 1225819 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 1225819 WARN  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1225819 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 1225819 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36844622650321657]
   [junit4]   2> 1225820 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@569f4e3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74c9f4df),segFN=segments_1,generation=1}
   [junit4]   2> 1225820 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1225845 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1225848 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1225849 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1225850 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/./collection1/data/
   [junit4]   2> 1225851 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1225852 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1225858 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1225858 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: lazy,/admin/ping,/admin/properties,/update,/admin/luke,/admin/system,/admin/mbeans,/config,/update/json/docs,/admin/threads,/admin/plugins,/replication,/schema,/get,/update/json,standard,/admin/file,/update/csv,defaults,/admin/logging,/admin/segments
   [junit4]   2> 1225863 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1225864 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1225864 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1225866 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@790ec6e7
   [junit4]   2> 1225866 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@569f4e3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74c9f4df),segFN=segments_1,generation=1}
   [junit4]   2> 1225866 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1225867 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@15000ac7[collection1] main
   [junit4]   2> 1225867 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1/conf
   [junit4]   2> 1225867 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1225867 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-001/collection1/conf
   [junit4]   2> 1225867 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1225868 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1225868 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1225868 INFO  (coreLoadExecutor-4439-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1225880 INFO  (searcherExecutor-4440-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@15000ac7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1226246 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1
   [junit4]   2> 1226249 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1226271 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a046505{/solr,null,AVAILABLE}
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.ServerConnector Started ServerConnector@26843066{HTTP/1.1}{127.0.0.1:39777}
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server Started @1229398ms
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/data, hostPort=39777, hostContext=/solr}
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4f6de641
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002'
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1226272 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1226273 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/solr.xml
   [junit4]   2> 1226317 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/.
   [junit4]   2> 1226318 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer New CoreContainer 443998018
   [junit4]   2> 1226318 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002]
   [junit4]   2> 1226318 WARN  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/lib
   [junit4]   2> 1226321 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1226326 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1226327 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1226327 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1226327 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1226327 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1226328 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/.
   [junit4]   2> 1226338 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, dataDir=data/, loadOnStartup=true, transient=false, configSetProperties=configsetprops.json, config=solrconfig.xml, schema=schema.xml}
   [junit4]   2> 1226339 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1
   [junit4]   2> 1226339 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1226358 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1'
   [junit4]   2> 1226359 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1226359 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1226359 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1226359 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1226367 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1226494 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1226584 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1226600 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1226617 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1226621 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1'
   [junit4]   2> 1226621 INFO  (coreLoadExecutor-4449-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1
   [junit4]   2> 1226621 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1226622 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1], dataDir=[null]
   [junit4]   2> 1226622 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1226625 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data
   [junit4]   2> 1226625 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 1226625 WARN  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1226626 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1226626 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36844622650321657]
   [junit4]   2> 1226629 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@19dd431f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d61b57d),segFN=segments_1,generation=1}
   [junit4]   2> 1226629 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1226643 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1226645 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226646 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226647 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226648 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226649 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/
   [junit4]   2> 1226659 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1226659 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: lazy,/admin/ping,/admin/properties,/update,/admin/luke,/admin/system,/admin/mbeans,/config,/update/json/docs,/admin/threads,/admin/plugins,/replication,/schema,/get,/update/json,standard,/admin/file,/update/csv,defaults,/admin/logging,/admin/segments
   [junit4]   2> 1226660 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1226667 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1226667 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1226667 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4746c9ee
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@19dd431f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d61b57d),segFN=segments_1,generation=1}
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@59b82977[collection1] main
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1226668 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf
   [junit4]   2> 1226669 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1226671 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1226671 INFO  (coreLoadExecutor-4449-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1226672 INFO  (searcherExecutor-4450-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@59b82977[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1226866 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@26843066{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1226866 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=443998018
   [junit4]   2> 1226866 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@661bb296
   [junit4]   2> 1226867 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1226867 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 1226867 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1226867 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1226870 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 1226870 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1226870 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index;done=false>>]
   [junit4]   2> 1226870 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1226870 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data;done=false>>]
   [junit4]   2> 1226871 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data
   [junit4]   2> 1226872 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    x:collection1] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@a046505{/solr,null,UNAVAILABLE}
   [junit4]   2> 1226874 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1226877 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b6329e9{/solr,null,AVAILABLE}
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.ServerConnector Started ServerConnector@1b5485d6{HTTP/1.1}{127.0.0.1:32879}
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server Started @1230004ms
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=32879, hostContext=/solr, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/data}
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4f6de641
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002'
   [junit4]   2> 1226878 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1226879 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1226879 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/solr.xml
   [junit4]   2> 1226890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/.
   [junit4]   2> 1226890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer New CoreContainer 1045830494
   [junit4]   2> 1226890 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002]
   [junit4]   2> 1226891 WARN  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/lib
   [junit4]   2> 1226892 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1226893 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1226893 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1226893 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1226893 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1226893 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1226894 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/.
   [junit4]   2> 1226895 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {transient=false, config=solrconfig.xml, dataDir=data/, name=collection1, configSetProperties=configsetprops.json, schema=schema.xml, loadOnStartup=true}
   [junit4]   2> 1226895 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1
   [junit4]   2> 1226895 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1226895 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1'
   [junit4]   2> 1226897 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1226897 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1226905 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1226905 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1226907 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1226920 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1226944 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1226949 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1226956 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1226958 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1'
   [junit4]   2> 1226958 INFO  (coreLoadExecutor-4460-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1
   [junit4]   2> 1226959 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1226959 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1], dataDir=[null]
   [junit4]   2> 1226959 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1226960 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data
   [junit4]   2> 1226960 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 1226960 WARN  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1226960 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1226960 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36844622650321657]
   [junit4]   2> 1226961 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@245d75ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@73e0336c),segFN=segments_1,generation=1}
   [junit4]   2> 1226961 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1226963 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1226966 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/./collection1/data/
   [junit4]   2> 1226966 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226967 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226969 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226970 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1226971 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1226971 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: lazy,/admin/ping,/admin/properties,/update,/admin/luke,/admin/system,/admin/mbeans,/config,/update/json/docs,/admin/threads,/admin/plugins,/replication,/schema,/get,/update/json,standard,/admin/file,/update/csv,defaults,/admin/logging,/admin/segments
   [junit4]   2> 1226972 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1226973 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1226973 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1226973 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7d1798fb
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@245d75ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@73e0336c),segFN=segments_1,generation=1}
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@3e263278[collection1] main
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-002/collection1/conf
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1226974 INFO  (coreLoadExecutor-4460-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1226975 INFO  (searcherExecutor-4461-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3e263278[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1227406 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1
   [junit4]   2> 1227411 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1227418 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@34d9a745{/solr,null,AVAILABLE}
   [junit4]   2> 1227418 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3d74be1e{HTTP/1.1}{127.0.0.1:49434}
   [junit4]   2> 1227418 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.e.j.s.Server Started @1230544ms
   [junit4]   2> 1227418 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1/data, hostContext=/solr, hostPort=49434}
   [junit4]   2> 1227419 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4f6de641
   [junit4]   2> 1227419 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003'
   [junit4]   2> 1227419 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1227419 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1227419 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/solr.xml
   [junit4]   2> 1227431 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/.
   [junit4]   2> 1227431 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer New CoreContainer 1316711039
   [junit4]   2> 1227431 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003]
   [junit4]   2> 1227431 WARN  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/lib
   [junit4]   2> 1227433 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1227434 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1227434 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1227434 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1227434 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1227434 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1227435 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/.
   [junit4]   2> 1227436 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, loadOnStartup=true, transient=false, schema=schema.xml, configSetProperties=configsetprops.json, config=solrconfig.xml, dataDir=data/}
   [junit4]   2> 1227436 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1
   [junit4]   2> 1227436 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1227437 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1'
   [junit4]   2> 1227444 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1227444 INFO  (TEST-TestReplicationHandler.doTestRepeater-seed#[DA5873B6F3FA9AF8]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1227444 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1227444 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1227454 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1227475 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1227517 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1227521 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1227528 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1227530 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1'
   [junit4]   2> 1227530 INFO  (coreLoadExecutor-4470-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1
   [junit4]   2> 1227530 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1227530 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1], dataDir=[null]
   [junit4]   2> 1227530 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1227531 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data
   [junit4]   2> 1227531 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data/index/
   [junit4]   2> 1227532 WARN  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1227532 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data/index
   [junit4]   2> 1227532 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.36844622650321657]
   [junit4]   2> 1227532 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@191f41b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5313a3ea),segFN=segments_1,generation=1}
   [junit4]   2> 1227532 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1227534 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data/
   [junit4]   2> 1227536 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1227538 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1227539 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1227540 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1227541 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1227542 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1227543 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: lazy,/admin/ping,/admin/properties,/update,/admin/luke,/admin/system,/admin/mbeans,/config,/update/json/docs,/admin/threads,/admin/plugins,/replication,/schema,/get,/update/json,standard,/admin/file,/update/csv,defaults,/admin/logging,/admin/segments
   [junit4]   2> 1227544 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1227544 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1227544 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1227544 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6f0d3507
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@191f41b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5313a3ea),segFN=segments_1,generation=1}
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@5be8056f[collection1] main
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1/conf
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1/conf}
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/collection1/conf
   [junit4]   2> 1227545 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1227546 WARN  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1227547 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1227547 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml
   [junit4]   2> 1227547 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1227547 INFO  (coreLoadExecutor-4470-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1227547 INFO  (searcherExecutor-4471-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5be8056f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1227950 INFO  (qtp755704053-11237) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 0
   [junit4]   2> 1227953 INFO  (qtp755704053-11236) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 0
   [junit4]   2> 1227955 INFO  (qtp755704053-11238) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 0
   [junit4]   2> 1227957 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1227959 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@569f4e3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74c9f4df),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@569f4e3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74c9f4df),segFN=segments_2,generation=2}
   [junit4]   2> 1227960 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1227962 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@2556203e[collection1] main
   [junit4]   2> 1227962 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1227962 INFO  (searcherExecutor-4440-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2556203e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.5.0):C3)))}
   [junit4]   2> 1227962 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp=/solr path=/update params={waitSearcher=true&softCommit=false&commit=true&wt=javabin&version=2} {commit=} 0 4
   [junit4]   2> 1227966 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1227972 INFO  (qtp755704053-11240) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
   [junit4]   2> 1227972 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 1227972 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1227972 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1227977 INFO  (qtp755704053-11237) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={command=filelist&qt=/replication&generation=2&wt=javabin&version=2} status=0 QTime=2 
   [junit4]   2> 1227977 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 10
   [junit4]   2> 1227980 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/solr-instance-003/./collection1/data/index.20151207024828861
   [junit4]   2> 1227980 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@191f41b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5313a3ea),segFN=segments_1,generation=1}
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@191f41b8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5313a3ea),segFN=segments_1,generation=1}
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1227982 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_DA5873B6F3FA9AF8-001/index-NIOFSDirectory-005 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a3dbf04; maxCacheMB=0.0126953125 maxMergeSizeMB=0.09375))
   [junit4]   2> 1227995 INFO  (qtp755704053-11236) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={checksum=true&file=_0.fdx&command=filecontent&qt=/replication&generation=2&wt=filestream} status=0 QTime=3 
   [junit4]   2> 1228000 INFO  (qtp755704053-11238) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={checksum=true&file=_0.nvd&command=filecontent&qt=/replication&generation=2&wt=filestream} status=0 QTime=0 
   [junit4]   2> 1228004 INFO  (qtp755704053-11239) [    x:collection1] o.a.s.c.S.Request [collection1] webapp=/solr path=/replication params={checksum=true&file=_0_Lucene50_0.doc&command=filecontent&qt=/replication&generation=2&wt=filestream} status=0 QTime=0 
   [junit4]   2> 1228007 INFO  (qtp755704053

[...truncated too long message...]

/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/analysis/icu/lucene-analyzers-icu-5.5.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.1.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J2-20151206_152757_655.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J2-20151206_152757_655.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1503)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:130)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:963)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:960)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:799: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:736: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 221 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1037 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1037/

2 tests failed.
FAILED:  org.apache.lucene.search.TestGeoPointQuery.testRandomBig

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([9F451249890883AF]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestGeoPointQuery

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([9F451249890883AF]:0)




Build Log:
[...truncated 8703 lines...]
   [junit4] Suite: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2> ??? 05, 2015 1:58:23 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:244)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:355)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
   [junit4]   2>    2) Thread[id=38, name=T2, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Long.valueOf(Long.java:577)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.mortonHash(GeoUtils.java:50)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.radiusQueryCanBeWrong(TestGeoPointQuery.java:206)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.distanceRangeContainsPoint(TestGeoPointQuery.java:192)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2$2.shouldMatch(BaseGeoPointTestCase.java:701)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:526)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    3) Thread[id=8, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2>    4) Thread[id=16, name=SUITE-TestGeoPointQuery-seed#[9F451249890883AF], state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1589)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:688)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:685)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:685)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:701)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
   [junit4]   2>    5) Thread[id=17, name=TEST-TestGeoPointQuery.testRandomBig-seed#[9F451249890883AF], state=WAITING, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:764)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    6) Thread[id=36, name=T0, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesPoly(GeoUtils.java:208)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectWithinPoly(GeoUtils.java:329)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellWithin(GeoPointInPolygonQuery.java:156)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    7) Thread[id=37, name=T1, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesPoly(GeoUtils.java:208)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectWithinPoly(GeoUtils.java:329)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellWithin(GeoPointInPolygonQuery.java:156)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:652)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    8) Thread[id=39, name=T3, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:109)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.<init>(GeoPointDistanceQueryImpl.java:58)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl.getTermsEnum(GeoPointDistanceQueryImpl.java:47)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:652)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:198)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:239)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.method=testRandomBig -Dtests.seed=9F451249890883AF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=el_CY -Dtests.timezone=America/Argentina/Jujuy -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   7416s J1 | TestGeoPointQuery.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9F451249890883AF]:0)
   [junit4]   2> ??? 05, 2015 2:18:43 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 5 leaked thread(s).
   [junit4]   2> ??? 05, 2015 2:28:02 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 5 threads leaked from SUITE scope at org.apache.lucene.search.TestGeoPointQuery: 
   [junit4]   2>    1) Thread[id=17, name=TEST-TestGeoPointQuery.testRandomBig-seed#[9F451249890883AF], state=WAITING, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:764)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    2) Thread[id=36, name=T0, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesPoly(GeoUtils.java:208)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectWithinPoly(GeoUtils.java:329)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellWithin(GeoPointInPolygonQuery.java:156)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    3) Thread[id=37, name=T1, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesPoly(GeoUtils.java:208)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectWithinPoly(GeoUtils.java:329)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellWithin(GeoPointInPolygonQuery.java:156)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:104)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:652)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    4) Thread[id=39, name=T3, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:109)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.<init>(GeoPointDistanceQueryImpl.java:58)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl.getTermsEnum(GeoPointDistanceQueryImpl.java:47)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:198)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:239)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    5) Thread[id=38, name=T2, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Long.valueOf(Long.java:577)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.mortonHash(GeoUtils.java:50)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.radiusQueryCanBeWrong(TestGeoPointQuery.java:203)
   [junit4]   2>         at org.apache.lucene.search.TestGeoPointQuery.distanceRangeContainsPoint(TestGeoPointQuery.java:192)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2$2.shouldMatch(BaseGeoPointTestCase.java:701)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:526)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2> ??? 05, 2015 2:30:42 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=17, name=TEST-TestGeoPointQuery.testRandomBig-seed#[9F451249890883AF], state=WAITING, group=TGRP-TestGeoPointQuery]
   [junit4]   2>    2) Thread[id=36, name=T0, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>    3) Thread[id=37, name=T1, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2>    4) Thread[id=39, name=T3, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>    5) Thread[id=38, name=T2, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2> ??? 05, 2015 2:36:12 ?? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=38, name=T2, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.lang.Integer.valueOf(Integer.java:642)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:521)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    2) Thread[id=17, name=SUITE-TestGeoPointQuery-seed#[9F451249890883AF]-worker, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1403)
   [junit4]   2>         at java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1396)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ClassModel.getAnnotatedLeafMethods(ClassModel.java:254)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.getShuffledMethods(RandomizedRunner.java:1254)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$500(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:832)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    3) Thread[id=36, name=T0, state=BLOCKED, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesPoly(GeoUtils.java:208)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellCrosses(GeoPointInPolygonQuery.java:150)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.cellIntersectsShape(GeoPointInPolygonQuery.java:162)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:105)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    4) Thread[id=37, name=T1, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:111)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery$GeoPolygonTermsEnum.<init>(GeoPointInPolygonQuery.java:145)
   [junit4]   2>         at org.apache.lucene.search.GeoPointInPolygonQuery.getTermsEnum(GeoPointInPolygonQuery.java:84)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.cache(LRUQueryCache.java:604)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:652)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2>    5) Thread[id=39, name=T3, state=RUNNABLE, group=TGRP-TestGeoPointQuery]
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.isClosestPointOnRectWithinRange(GeoUtils.java:368)
   [junit4]   2>         at org.apache.lucene.util.GeoUtils.rectCrossesCircle(GeoUtils.java:362)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.cellCrosses(GeoPointDistanceQueryImpl.java:80)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.cellIntersectsShape(GeoPointDistanceQueryImpl.java:90)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:105)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:84)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.relateAndRecurse(GeoPointTermsEnum.java:114)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.computeRange(GeoPointTermsEnum.java:83)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermsEnum.<init>(GeoPointTermsEnum.java:64)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl$GeoPointRadiusTermsEnum.<init>(GeoPointDistanceQueryImpl.java:58)
   [junit4]   2>         at org.apache.lucene.search.GeoPointDistanceQueryImpl.getTermsEnum(GeoPointDistanceQueryImpl.java:47)
   [junit4]   2>         at org.apache.lucene.search.MultiTermQuery.getTermsEnum(MultiTermQuery.java:318)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.getDocIDs(GeoPointTermQueryConstantScoreWrapper.java:72)
   [junit4]   2>         at org.apache.lucene.search.GeoPointTermQueryConstantScoreWrapper$1.bulkScorer(GeoPointTermQueryConstantScoreWrapper.java:117)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.booleanScorer(BooleanWeight.java:198)
   [junit4]   2>         at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:239)
   [junit4]   2>         at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.bulkScorer(LRUQueryCache.java:655)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.AssertingWeight.bulkScorer(AssertingWeight.java:69)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:818)
   [junit4]   2>         at org.apache.lucene.search.AssertingIndexSearcher.search(AssertingIndexSearcher.java:92)
   [junit4]   2>         at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$VerifyHits.test(BaseGeoPointTestCase.java:496)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2._run(BaseGeoPointTestCase.java:753)
   [junit4]   2>         at org.apache.lucene.util.BaseGeoPointTestCase$2.run(BaseGeoPointTestCase.java:618)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {id=FSTOrd50, point=PostingsFormat(name=Direct)}, docValues:{id=DocValuesFormat(name=Memory), point=DocValuesFormat(name=Direct)}, sim=DefaultSimilarity, locale=el_CY, timezone=America/Argentina/Jujuy
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=6,free=38789464,total=463470592
   [junit4]   2> NOTE: All tests run in this JVM: [DuplicateFilterTest, TestGeoUtils, TestGeoPointQuery]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.seed=9F451249890883AF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=el_CY -Dtests.timezone=America/Argentina/Jujuy -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestGeoPointQuery (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9F451249890883AF]:0)
   [junit4] Completed [20/20] on J1 in 10245.45s, 23 tests, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:799: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:736: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:472: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2248: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 20 suites, 142 tests, 1 suite-level error, 1 error, 1 ignored (1 assumption) [seed: 9F451249890883AF]

Total time: 291 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1036 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1036/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=32047, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=32047, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:48527/kn_u: Could not find collection : awholynewstresscollection_collection4_1
	at __randomizedtesting.SeedInfo.seed([42AE944646D3DDD0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:888)


FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([42AE944646D3DDD0:9C24FA3B7D71BF23]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:175)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:136)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:831)
	at org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:104)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10717 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/init-core-data-001
   [junit4]   2> 1001602 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[42AE944646D3DDD0]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1001604 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[42AE944646D3DDD0]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /kn_u/
   [junit4]   2> 1001618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1001618 INFO  (Thread-25889) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1001618 INFO  (Thread-25889) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1001718 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkTestServer start zk server on port:39818
   [junit4]   2> 1001719 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1001720 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1001724 INFO  (zkCallback-1800-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1878258a name:ZooKeeperConnection Watcher:127.0.0.1:39818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1001724 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1001724 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1001724 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1001728 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1001732 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1001732 INFO  (zkCallback-1801-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@53db2de9 name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1001732 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1001733 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1001733 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1001735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1001737 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1001739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1001741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1001741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1001743 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1001744 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1001745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1001746 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1001747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1001747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1001749 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1001749 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1001751 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1001751 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1001753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1001754 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1001757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1001757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1001758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1001758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1001760 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1001760 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1001761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1001761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1002076 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1
   [junit4]   2> 1002077 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1002081 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2dff72ad{/kn_u,null,AVAILABLE}
   [junit4]   2> 1002083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.ServerConnector Started ServerConnector@5dc1d372{HTTP/1.1}{127.0.0.1:48527}
   [junit4]   2> 1002083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.Server Started @1005020ms
   [junit4]   2> 1002083 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=48527, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/tempDir-001/control/data, hostContext=/kn_u, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores}
   [junit4]   2> 1002085 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 1002085 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001'
   [junit4]   2> 1002085 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1002085 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1002086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1002088 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1002088 INFO  (zkCallback-1802-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@70b6f9a8 name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1002088 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1002089 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1002090 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1002090 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/solr.xml
   [junit4]   2> 1002105 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores
   [junit4]   2> 1002105 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer New CoreContainer 1194329061
   [junit4]   2> 1002105 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001]
   [junit4]   2> 1002106 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/lib
   [junit4]   2> 1002107 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1002112 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1002113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1002113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1002113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39818/solr
   [junit4]   2> 1002113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1002113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1002115 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1002116 INFO  (zkCallback-1804-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@567d5ff0 name:ZooKeeperConnection Watcher:127.0.0.1:39818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1002117 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1002117 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1002119 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1002120 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@57fe7b3b name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1002120 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1002122 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1002124 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1002127 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1002129 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1002132 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1002134 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1002135 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1002137 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1002138 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1002139 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48527_kn_u
   [junit4]   2> 1002140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:48527_kn_u
   [junit4]   2> 1002141 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1002143 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1002144 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1002145 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94978216905998340-127.0.0.1:48527_kn_u-n_0000000000
   [junit4]   2> 1002145 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:48527_kn_u
   [junit4]   2> 1002145 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1002146 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer Overseer (id=94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) starting
   [junit4]   2> 1002148 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1002155 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1002156 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1002156 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1002167 INFO  (OverseerCollectionConfigSetProcessor-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1002170 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1002199 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1002199 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1002200 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores
   [junit4]   2> 1002201 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {collection=control_collection, coreNodeName=, schema=schema.xml, shard=, name=collection1, configSetProperties=configsetprops.json, dataDir=data/, loadOnStartup=true, config=solrconfig.xml, transient=false}
   [junit4]   2> 1002201 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1
   [junit4]   2> 1002201 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1002202 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1002202 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1002202 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 1002202 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:48527_kn_u    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1002205 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:48527_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:48527/kn_u",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null} current state version: 0
   [junit4]   2> 1002205 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1002206 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:48527_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:48527/kn_u",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null}
   [junit4]   2> 1002206 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1002206 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1002208 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1002209 INFO  (zkCallback-1805-thread-2-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1003206 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1003206 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1003207 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1003208 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1003209 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1003209 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1'
   [junit4]   2> 1003209 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1003209 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1003211 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1003221 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1003234 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1003278 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1003287 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1003532 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1003540 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1003542 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1003553 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1003556 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1003558 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1003559 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1003559 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1003559 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1003560 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1003560 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1003561 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 1003561 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1003561 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1003562 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1], dataDir=[null]
   [junit4]   2> 1003562 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@91d39bf
   [junit4]   2> 1003562 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data
   [junit4]   2> 1003562 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1003563 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1003563 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data/index
   [junit4]   2> 1003563 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=81.46875, floorSegmentMB=1.2275390625, forceMergeDeletesPctAllowed=27.03868587713784, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1003564 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7fe7f867 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@566573cb),segFN=segments_1,generation=1}
   [junit4]   2> 1003564 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1003564 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data/
   [junit4]   2> 1003564 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1003569 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1003570 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1003570 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1003570 WARN  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1003573 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1003574 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1003575 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1003576 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1003580 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/admin/threads,/update/json,/admin/luke,/update/json/docs,/update,/admin/system,/dump,/admin/segments,/admin/logging,/replication,/admin/plugins,/schema,standard,/admin/properties,/config,/admin/ping,/admin/file,/get,/update/csv
   [junit4]   2> 1003581 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1003582 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1003582 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1003582 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1003582 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1003583 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1273988799, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1003583 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7fe7f867 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@566573cb),segFN=segments_1,generation=1}
   [junit4]   2> 1003583 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1003584 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@1511f65d[collection1] main
   [junit4]   2> 1003584 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/control_collection
   [junit4]   2> 1003585 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1003585 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1003585 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1003585 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1003585 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1003586 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1003586 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1003586 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1003586 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1003586 INFO  (searcherExecutor-2584-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1511f65d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1003587 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1003587 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1003587 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1003587 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1519651472558522368
   [junit4]   2> 1003589 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1519651472558522368
   [junit4]   2> 1003589 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1003590 INFO  (coreLoadExecutor-2583-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1003590 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:48527/kn_u collection:control_collection shard:shard1
   [junit4]   2> 1003591 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1003597 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/94978216905998340-core_node1-n_0000000000
   [junit4]   2> 1003597 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 1003600 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1003600 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1003600 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:48527/kn_u/collection1/
   [junit4]   2> 1003600 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1003600 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:48527/kn_u/collection1/ has no replicas
   [junit4]   2> 1003600 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1003601 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1003604 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1003606 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:48527/kn_u/collection1/ shard1
   [junit4]   2> 1003606 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:48527/kn_u",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1003705 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1003705 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1003707 INFO  (zkCallback-1807-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4be2b750 name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1003707 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1003708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1003708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1003709 INFO  (zkCallback-1805-thread-2-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1003709 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1003709 INFO  (zkCallback-1807-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1003712 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1003757 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:48527/kn_u/collection1/ and leader is http://127.0.0.1:48527/kn_u/collection1/
   [junit4]   2> 1003757 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48527/kn_u
   [junit4]   2> 1003757 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1003757 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1003757 INFO  (coreZkRegister-2577-thread-1-processing-x:collection1 n:127.0.0.1:48527_kn_u c:control_collection r:core_node1 s:shard1) [n:127.0.0.1:48527_kn_u c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1003759 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:48527_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:48527/kn_u",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":"shard1"} current state version: 2
   [junit4]   2> 1003760 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:48527_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:48527/kn_u",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":"shard1"}
   [junit4]   2> 1003863 INFO  (zkCallback-1805-thread-2-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1003863 INFO  (zkCallback-1807-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1003863 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1004006 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores/collection1
   [junit4]   2> 1004007 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001
   [junit4]   2> 1004008 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 1004010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f47336d{/kn_u,null,AVAILABLE}
   [junit4]   2> 1004011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.ServerConnector Started ServerConnector@770422d9{HTTP/1.1}{127.0.0.1:34048}
   [junit4]   2> 1004011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.e.j.s.Server Started @1006948ms
   [junit4]   2> 1004011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostPort=34048, solrconfig=solrconfig.xml, hostContext=/kn_u, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/tempDir-001/jetty1, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores}
   [junit4]   2> 1004011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@7b3cb2c6
   [junit4]   2> 1004011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001'
   [junit4]   2> 1004012 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1004012 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1004012 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1004013 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1004014 INFO  (zkCallback-1808-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@371fbf99 name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1004014 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1004014 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1004015 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1004015 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/solr.xml
   [junit4]   2> 1004029 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores
   [junit4]   2> 1004030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer New CoreContainer 456153429
   [junit4]   2> 1004030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001]
   [junit4]   2> 1004030 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/lib
   [junit4]   2> 1004030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39818/solr
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1004032 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1004033 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1004034 INFO  (zkCallback-1810-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4411ef82 name:ZooKeeperConnection Watcher:127.0.0.1:39818 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1004036 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1004036 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1004039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1004040 INFO  (zkCallback-1811-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4ec1a51 name:ZooKeeperConnection Watcher:127.0.0.1:39818/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1004040 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1004045 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1005050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34048_kn_u
   [junit4]   2> 1005051 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34048_kn_u
   [junit4]   2> 1005052 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1005052 INFO  (zkCallback-1811-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1005053 INFO  (zkCallback-1807-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1005052 INFO  (zkCallback-1805-thread-2-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1005053 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1005055 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94978216905998344-127.0.0.1:34048_kn_u-n_0000000001
   [junit4]   2> 1005056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/94978216905998340-127.0.0.1:48527_kn_u-n_0000000000 to know if I could be the leader
   [junit4]   2> 1005066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1005066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1005067 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores
   [junit4]   2> 1005068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {collection=collection1, coreNodeName=, configSetProperties=configsetprops.json, dataDir=data/, transient=false, schema=schema.xml, name=collection1, shard=, loadOnStartup=true, config=solrconfig.xml}
   [junit4]   2> 1005068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores/collection1
   [junit4]   2> 1005068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1005069 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1005069 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
   [junit4]   2> 1005070 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[42AE944646D3DDD0]) [n:127.0.0.1:34048_kn_u    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1005069 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1005072 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:34048_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:34048/kn_u",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null} current state version: 3
   [junit4]   2> 1005072 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1005072 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "node_name":"127.0.0.1:34048_kn_u",
   [junit4]   2>   "base_url":"http://127.0.0.1:34048/kn_u",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "shard":null}
   [junit4]   2> 1005072 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1005072 INFO  (OverseerStateUpdate-94978216905998340-127.0.0.1:48527_kn_u-n_0000000000) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1005175 INFO  (zkCallback-1805-thread-1-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1005175 INFO  (zkCallback-1811-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1005175 INFO  (zkCallback-1805-thread-2-processing-n:127.0.0.1:48527_kn_u) [n:127.0.0.1:48527_kn_u    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1005175 INFO  (zkCallback-1807-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1006073 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1006073 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1006074 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1006074 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:/collections/collection1
   [junit4]   2> 1006075 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1006075 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_42AE944646D3DDD0-001/shard-1-001/cores/collection1'
   [junit4]   2> 1006075 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1006076 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1006078 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1006086 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1006099 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
   [junit4]   2> 1006127 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1006136 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1006392 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1006400 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1006402 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1006412 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1006417 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1006419 INFO  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1006420 WARN  (coreLoadExecutor-2594-thread-1-processing-n:127.0.0.1:34048_kn_u) [n:127.0.0.1:34048_kn_u c:collection1   x:collection1] o.a.s.s.OpenExchang

[...truncated too long message...]

9
   [junit4]   2> 3548190 WARN  (Thread-47272) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	11	/solr/aliases.json
   [junit4]   2> 	10	/solr/security.json
   [junit4]   2> 	10	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	17	/solr/collections/collection1/state.json
   [junit4]   2> 	11	/solr/clusterstate.json
   [junit4]   2> 	9	/solr/collections/all_in_lir/state.json
   [junit4]   2> 	6	/solr/overseer_elect/election/94978356697956356-127.0.0.1:34738_zhis%2Fgm-n_0000000000
   [junit4]   2> 	2	/solr/overseer_elect/election/94978356697956377-127.0.0.1:50715_zhis%2Fgm-n_0000000007
   [junit4]   2> 	2	/solr/collections/collection1/leader_elect/shard1/election/94978356697956377-core_node3-n_0000000006
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	12	/solr/live_nodes
   [junit4]   2> 	11	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderInitiatedRecoveryOnShardRestartTest -Dtests.method=testRestartWithAllInLIR -Dtests.seed=42AE944646D3DDD0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ca -Dtests.timezone=US/Mountain -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  413s J2 | LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([42AE944646D3DDD0:9C24FA3B7D71BF23]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:175)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:136)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:831)
   [junit4]    > 	at org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:104)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3548193 INFO  (SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[42AE944646D3DDD0]-worker) [n:127.0.0.1:50715_zhis%2Fgm c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_42AE944646D3DDD0-001
   [junit4]   2> Dec 04, 2015 6:52:03 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=28343, maxDocsPerChunk=9, blockSize=516), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=28343, blockSize=516)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ca, timezone=US/Mountain
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=109854896,total=525860864
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressRecovery, RegexBoostProcessorTest, TestSolrIndexConfig, TestConfigSetsAPIZkFailure, DeleteReplicaTest, SolrCoreCheckLockOnStartupTest, TestCoreDiscovery, SecurityConfHandlerTest, TestFastWriter, TestCSVResponseWriter, TestDistribIDF, HdfsBasicDistributedZkTest, SuggesterWFSTTest, DistributedFacetPivotLongTailTest, TestReloadDeadlock, CurrencyFieldXmlFileTest, RecoveryZkTest, BadIndexSchemaTest, NoCacheHeaderTest, StatelessScriptUpdateProcessorFactoryTest, LukeRequestHandlerTest, QueryParsingTest, TestHighlightDedupGrouping, TestExtendedDismaxParser, TestJettySolrRunner, TestReload, TestManagedStopFilterFactory, LeaderElectionTest, VersionInfoTest, DistribCursorPagingTest, TestSearchPerf, DocExpirationUpdateProcessorFactoryTest, SearchHandlerTest, OverseerRolesTest, TestExactStatsCache, TestNonDefinedSimilarityFactory, TestDistributedStatsComponentCardinality, HdfsThreadLeakTest, DistanceUnitsTest, TestDistributedMissingSort, TestCollapseQParserPlugin, DistribDocExpirationUpdateProcessorTest, RankQueryTest, TestCloudManagedSchema, TestPostingsSolrHighlighter, TestIndexSearcher, BigEndianAscendingWordSerializerTest, XmlUpdateRequestHandlerTest, TestManagedSchemaFieldTypeResource, TestZkChroot, TestRuleBasedAuthorizationPlugin, TestStandardQParsers, RecoveryAfterSoftCommitTest, TestSolrDeletionPolicy1, FacetPivotSmallTest, TestUtils, TestOverriddenPrefixQueryForCustomFieldType, RollingRestartTest, TestWriterPerf, TestRestManager, TestSha256AuthenticationProvider, SuggesterTest, SolrXmlInZkTest, BadComponentTest, SaslZkACLProviderTest, TestGroupingSearch, TestPHPSerializedResponseWriter, TestXIncludeConfig, TestReplicationHandlerBackup, TestSolrQueryParserResource, BJQParserTest, TestRecovery, BufferStoreTest, PreAnalyzedUpdateProcessorTest, FullSolrCloudDistribCmdsTest, TestFreeTextSuggestions, TestRequestForwarding, TestExpandComponent, QueryResultKeyTest, SampleTest, RequestLoggingTest, DirectUpdateHandlerOptimizeTest, OverseerTest, CoreAdminHandlerTest, ClusterStateTest, TestIndexingPerformance, TestJmxMonitoredMap, HighlighterMaxOffsetTest, TestClusterStateMutator, AsyncCallRequestStatusResponseTest, ShardRoutingTest, BasicZkTest, TestReplicationHandler, AutoCommitTest, TestFaceting, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, DistributedTermsComponentTest, SolrCoreTest, PeerSyncTest, ConvertedLegacyTest, TestSort, TestFiltering, TestFunctionQuery, TestLazyCores, DirectUpdateHandlerTest, SoftAutoCommitTest, SolrIndexSplitterTest, SuggesterFSTTest, TestFoldingMultitermQuery, SuggesterTSTTest, WordBreakSolrSpellCheckerTest, FieldMutatingUpdateProcessorTest, TestRemoteStreaming, DebugComponentTest, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TermsComponentTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, TestCollationField, UpdateRequestProcessorFactoryTest, MBeansHandlerTest, BinaryUpdateRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, SolrIndexConfigTest, TestSolrDeletionPolicy2, MultiTermTest, TestDocSet, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, RAMDirectoryFactoryTest, TestSolrJ, ZkNodePropsTest, SliceStateTest, FileUtilsTest, CircularListTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestSimpleTrackingShardHandler, AsyncMigrateRouteKeyTest, ConfigSetsAPITest, DeleteShardTest, LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4] Completed [439/552] on J2 in 413.76s, 1 test, 1 failure <<< FAILURES!

[...truncated 359 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:799: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:736: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 552 suites (7 ignored), 2185 tests, 1 error, 1 failure, 33 ignored (18 assumptions) [seed: 42AE944646D3DDD0]

Total time: 209 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1035 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1035/

2 tests failed.
FAILED:  org.apache.solr.hadoop.MorphlineBasicMiniMRTest.mrRun

Error Message:
Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "lucene1-us-west/10.41.0.5"; destination host is: "lucene1-us-west.apache.org":53477; 

Stack Trace:
java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "lucene1-us-west/10.41.0.5"; destination host is: "lucene1-us-west.apache.org":53477; 
	at __randomizedtesting.SeedInfo.seed([562707F3EC0A0F3D:5875B3FDED9C3D32]:0)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at com.sun.proxy.$Proxy109.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy110.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:461)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.getClusterMetrics(ResourceMgrDelegate.java:151)
	at org.apache.hadoop.mapred.YARNRunner.getClusterMetrics(YARNRunner.java:179)
	at org.apache.hadoop.mapreduce.Cluster.getClusterStatus(Cluster.java:246)
	at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:719)
	at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:717)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:717)
	at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:645)
	at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:608)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.solr.hadoop.MorphlineBasicMiniMRTest.mrRun(MorphlineBasicMiniMRTest.java:363)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:513)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)


FAILED:  org.apache.solr.hadoop.MorphlineGoLiveMiniMRTest.test

Error Message:
Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "lucene1-us-west/10.41.0.5"; destination host is: "lucene1-us-west.apache.org":50361; 

Stack Trace:
java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "lucene1-us-west/10.41.0.5"; destination host is: "lucene1-us-west.apache.org":50361; 
	at __randomizedtesting.SeedInfo.seed([562707F3EC0A0F3D:DE73382942F662C5]:0)
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
	at org.apache.hadoop.ipc.Client.call(Client.java:1472)
	at org.apache.hadoop.ipc.Client.call(Client.java:1399)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
	at com.sun.proxy.$Proxy111.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy112.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:461)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.getClusterMetrics(ResourceMgrDelegate.java:151)
	at org.apache.hadoop.mapred.YARNRunner.getClusterMetrics(YARNRunner.java:179)
	at org.apache.hadoop.mapreduce.Cluster.getClusterStatus(Cluster.java:246)
	at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:719)
	at org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:717)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:717)
	at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:645)
	at org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:608)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.solr.hadoop.MorphlineGoLiveMiniMRTest.test(MorphlineGoLiveMiniMRTest.java:402)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
	at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:513)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)




Build Log:
[...truncated 18504 lines...]
   [junit4] Suite: org.apache.solr.hadoop.MorphlineBasicMiniMRTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/init-core-data-001
   [junit4]   2> 0    INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1027 WARN  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1126 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem No KeyProvider found.
   [junit4]   2> 1168 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem fsLock is fair:true
   [junit4]   2> 1246 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.c.C.deprecation hadoop.configured.node.mapping is deprecated. Instead, use net.topology.configured.node.mapping
   [junit4]   2> 1246 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.DatanodeManager dfs.block.invalidate.limit=1000
   [junit4]   2> 1246 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.DatanodeManager dfs.namenode.datanode.registration.ip-hostname-check=true
   [junit4]   2> 1253 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
   [junit4]   2> 1256 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager The block deletion will start around 2015 ??? 03 23:01:28
   [junit4]   2> 1260 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map BlocksMap
   [junit4]   2> 1261 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 1263 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 2.0% max memory 455 MB = 9.1 MB
   [junit4]   2> 1263 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^20 = 1048576 entries
   [junit4]   2> 1293 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager dfs.block.access.token.enable=false
   [junit4]   2> 1294 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager defaultReplication         = 2
   [junit4]   2> 1294 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxReplication             = 512
   [junit4]   2> 1294 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager minReplication             = 1
   [junit4]   2> 1294 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxReplicationStreams      = 2
   [junit4]   2> 1295 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager shouldCheckForEnoughRacks  = false
   [junit4]   2> 1295 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager replicationRecheckInterval = 3000
   [junit4]   2> 1295 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager encryptDataTransfer        = false
   [junit4]   2> 1295 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxNumBlocksToLog          = 1000
   [junit4]   2> 1304 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem fsOwner             = jenkins (auth:SIMPLE)
   [junit4]   2> 1304 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem supergroup          = supergroup
   [junit4]   2> 1305 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem isPermissionEnabled = true
   [junit4]   2> 1305 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem HA Enabled: false
   [junit4]   2> 1312 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Append Enabled: true
   [junit4]   2> 1779 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map INodeMap
   [junit4]   2> 1779 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 1779 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 1.0% max memory 455 MB = 4.5 MB
   [junit4]   2> 1779 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^19 = 524288 entries
   [junit4]   2> 1780 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode Caching file names occuring more than 10 times
   [junit4]   2> 1796 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map cachedBlocks
   [junit4]   2> 1796 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 1796 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 0.25% max memory 455 MB = 1.1 MB
   [junit4]   2> 1796 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^17 = 131072 entries
   [junit4]   2> 1799 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.threshold-pct = 0.9990000128746033
   [junit4]   2> 1799 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.min.datanodes = 0
   [junit4]   2> 1799 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.extension     = 0
   [junit4]   2> 1802 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Retry cache on namenode is enabled
   [junit4]   2> 1802 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
   [junit4]   2> 1809 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map NameNodeRetryCache
   [junit4]   2> 1809 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 1809 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 0.029999999329447746% max memory 455 MB = 139.8 KB
   [junit4]   2> 1809 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^14 = 16384 entries
   [junit4]   2> 1820 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf ACLs enabled? false
   [junit4]   2> 1820 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf XAttrs enabled? true
   [junit4]   2> 1820 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf Maximum size of an xattr: 16384
   [junit4]   2> 1924 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSImage Allocated new BlockPoolId: BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 1943 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name1 has been successfully formatted.
   [junit4]   2> 1947 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name2 has been successfully formatted.
   [junit4]   2> 2187 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNStorageRetentionManager Going to retain 1 images with txid >= 0
   [junit4]   2> 2192 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode createNameNode []
   [junit4]   2> 2251 WARN  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2352 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.m.i.MetricsSystemImpl Scheduled snapshot period at 10 second(s).
   [junit4]   2> 2353 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.m.i.MetricsSystemImpl NameNode metrics system started
   [junit4]   2> 2356 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode fs.defaultFS is hdfs://127.0.0.1:0
   [junit4]   2> 2432 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.DFSUtil Starting Web-server for hdfs at: http://localhost:0
   [junit4]   2> 2599 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 2630 WARN  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2669 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
   [junit4]   2> 2673 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
   [junit4]   2> 2673 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
   [junit4]   2> 2674 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
   [junit4]   2> 2727 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter)
   [junit4]   2> 2730 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
   [junit4]   2> 2778 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Jetty bound to port 35720
   [junit4]   2> 2778 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2859 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_35720_hdfs____.cdvm0z/webapp
   [junit4]   2> 3151 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 5362 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35720
   [junit4]   2> 5378 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem No KeyProvider found.
   [junit4]   2> 5378 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem fsLock is fair:true
   [junit4]   2> 5379 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.DatanodeManager dfs.block.invalidate.limit=1000
   [junit4]   2> 5379 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.DatanodeManager dfs.namenode.datanode.registration.ip-hostname-check=true
   [junit4]   2> 5380 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
   [junit4]   2> 5380 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager The block deletion will start around 2015 ??? 03 23:01:32
   [junit4]   2> 5380 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map BlocksMap
   [junit4]   2> 5381 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 5381 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 2.0% max memory 455 MB = 9.1 MB
   [junit4]   2> 5381 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^20 = 1048576 entries
   [junit4]   2> 5382 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager dfs.block.access.token.enable=false
   [junit4]   2> 5382 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager defaultReplication         = 2
   [junit4]   2> 5382 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxReplication             = 512
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager minReplication             = 1
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxReplicationStreams      = 2
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager shouldCheckForEnoughRacks  = false
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager replicationRecheckInterval = 3000
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager encryptDataTransfer        = false
   [junit4]   2> 5383 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.b.BlockManager maxNumBlocksToLog          = 1000
   [junit4]   2> 5384 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem fsOwner             = jenkins (auth:SIMPLE)
   [junit4]   2> 5384 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem supergroup          = supergroup
   [junit4]   2> 5384 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem isPermissionEnabled = true
   [junit4]   2> 5385 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem HA Enabled: false
   [junit4]   2> 5385 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Append Enabled: true
   [junit4]   2> 5386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map INodeMap
   [junit4]   2> 5386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 5386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 1.0% max memory 455 MB = 4.5 MB
   [junit4]   2> 5386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^19 = 524288 entries
   [junit4]   2> 5387 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode Caching file names occuring more than 10 times
   [junit4]   2> 5387 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map cachedBlocks
   [junit4]   2> 5387 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 5387 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 0.25% max memory 455 MB = 1.1 MB
   [junit4]   2> 5388 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^17 = 131072 entries
   [junit4]   2> 5388 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.threshold-pct = 0.9990000128746033
   [junit4]   2> 5388 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.min.datanodes = 0
   [junit4]   2> 5388 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem dfs.namenode.safemode.extension     = 0
   [junit4]   2> 5388 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Retry cache on namenode is enabled
   [junit4]   2> 5389 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
   [junit4]   2> 5389 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet Computing capacity for map NameNodeRetryCache
   [junit4]   2> 5389 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet VM type       = 64-bit
   [junit4]   2> 5389 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet 0.029999999329447746% max memory 455 MB = 139.8 KB
   [junit4]   2> 5389 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.u.GSet capacity      = 2^14 = 16384 entries
   [junit4]   2> 5392 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf ACLs enabled? false
   [junit4]   2> 5392 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf XAttrs enabled? true
   [junit4]   2> 5393 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NNConf Maximum size of an xattr: 16384
   [junit4]   2> 5403 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name1/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 5410 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name2/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 5414 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FileJournalManager Recovering unfinalized segments in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name1/current
   [junit4]   2> 5415 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FileJournalManager Recovering unfinalized segments in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name2/current
   [junit4]   2> 5415 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSImage No edit log streams selected.
   [junit4]   2> 5452 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSImageFormatPBINode Loading 1 INodes.
   [junit4]   2> 5467 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSImageFormatProtobuf Loaded FSImage in 0 seconds.
   [junit4]   2> 5467 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSImage Loaded image for txid 0 from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/name1/current/fsimage_0000000000000000000
   [junit4]   2> 5474 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Need to save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
   [junit4]   2> 5475 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSEditLog Starting log segment at 1
   [junit4]   2> 5549 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameCache initialized with 0 entries 0 lookups
   [junit4]   2> 5549 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Finished loading FSImage in 156 msecs
   [junit4]   2> 5993 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode RPC server is binding to localhost:0
   [junit4]   2> 6008 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.i.CallQueueManager Using callQueue class java.util.concurrent.LinkedBlockingQueue
   [junit4]   2> 6035 INFO  (Socket Reader #1 for port 43259) [    ] o.a.h.i.Server Starting Socket Reader #1 for port 43259
   [junit4]   2> 6092 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode Clients are to use localhost:43259 to access this namenode/service.
   [junit4]   2> 6095 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Registered FSNamesystemState MBean
   [junit4]   2> 6123 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Number of blocks under construction: 0
   [junit4]   2> 6123 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Number of blocks under construction: 0
   [junit4]   2> 6123 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem initializing replication queues
   [junit4]   2> 6124 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.StateChange STATE* Leaving safe mode after 0 secs
   [junit4]   2> 6125 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.StateChange STATE* Network topology has 0 racks and 0 datanodes
   [junit4]   2> 6125 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.StateChange STATE* UnderReplicatedBlocks has 0 blocks
   [junit4]   2> 6131 INFO  (Replication Queue Initializer) [    ] o.a.h.h.s.b.BlockManager Total number of blocks            = 0
   [junit4]   2> 6132 INFO  (Replication Queue Initializer) [    ] o.a.h.h.s.b.BlockManager Number of invalid blocks          = 0
   [junit4]   2> 6132 INFO  (Replication Queue Initializer) [    ] o.a.h.h.s.b.BlockManager Number of under-replicated blocks = 0
   [junit4]   2> 6132 INFO  (Replication Queue Initializer) [    ] o.a.h.h.s.b.BlockManager Number of  over-replicated blocks = 0
   [junit4]   2> 6132 INFO  (Replication Queue Initializer) [    ] o.a.h.h.s.b.BlockManager Number of blocks being written    = 0
   [junit4]   2> 6132 INFO  (Replication Queue Initializer) [    ] o.a.h.h.StateChange STATE* Replication Queue initialization scan for invalid, over- and under-replicated blocks completed in 6 msec
   [junit4]   2> 6222 INFO  (IPC Server listener on 43259) [    ] o.a.h.i.Server IPC Server listener on 43259: starting
   [junit4]   2> 6222 INFO  (IPC Server Responder) [    ] o.a.h.i.Server IPC Server Responder: starting
   [junit4]   2> 6231 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.NameNode NameNode RPC up at: localhost/127.0.0.1:43259
   [junit4]   2> 6231 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.n.FSNamesystem Starting services required for active state
   [junit4]   2> 6255 INFO  (CacheReplicationMonitor(676048482)) [    ] o.a.h.h.s.b.CacheReplicationMonitor Starting CacheReplicationMonitor with interval 30000 milliseconds
   [junit4]   2> 6256 INFO  (CacheReplicationMonitor(676048482)) [    ] o.a.h.h.s.b.CacheReplicationMonitor Rescanning after 17636790587 milliseconds
   [junit4]   2> 6256 INFO  (CacheReplicationMonitor(676048482)) [    ] o.a.h.h.s.b.CacheReplicationMonitor Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s).
   [junit4]   2> 6258 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.MiniDFSCluster Starting DataNode 0 with dfs.datanode.data.dir: [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1,[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2
   [junit4]   2> 6344 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.m.i.MetricsSystemImpl DataNode metrics system started (again)
   [junit4]   2> 6346 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Configured hostname is 127.0.0.1
   [junit4]   2> 6354 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Starting DataNode with maxLockedMemory = 0
   [junit4]   2> 6364 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Opened streaming server at /127.0.0.1:34131
   [junit4]   2> 6368 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Balancing bandwith is 1048576 bytes/s
   [junit4]   2> 6368 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Number threads for balancing is 5
   [junit4]   2> 6383 WARN  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 6385 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
   [junit4]   2> 6386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
   [junit4]   2> 6386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
   [junit4]   2> 6386 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
   [junit4]   2> 6393 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.datanode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
   [junit4]   2> 6394 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Jetty bound to port 56245
   [junit4]   2> 6394 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 6417 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56245_datanode____.mzamnb/webapp
   [junit4]   2> 6604 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 7108 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56245
   [junit4]   2> 7110 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode dnUserName = jenkins
   [junit4]   2> 7110 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode supergroup = supergroup
   [junit4]   2> 7145 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.i.CallQueueManager Using callQueue class java.util.concurrent.LinkedBlockingQueue
   [junit4]   2> 7146 INFO  (Socket Reader #1 for port 34967) [    ] o.a.h.i.Server Starting Socket Reader #1 for port 34967
   [junit4]   2> 7163 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Opened IPC server at /127.0.0.1:34967
   [junit4]   2> 7177 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Refresh request received for nameservices: null
   [junit4]   2> 7181 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Starting BPOfferServices for nameservices: <default>
   [junit4]   2> 7209 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:43259 starting to offer service
   [junit4]   2> 7225 INFO  (IPC Server Responder) [    ] o.a.h.i.Server IPC Server Responder: starting
   [junit4]   2> 7234 INFO  (IPC Server listener on 34967) [    ] o.a.h.i.Server IPC Server listener on 34967: starting
   [junit4]   2> 7239 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.MiniDFSCluster Starting DataNode 1 with dfs.datanode.data.dir: [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3,[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4
   [junit4]   2> 7397 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.m.i.MetricsSystemImpl DataNode metrics system started (again)
   [junit4]   2> 7398 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Configured hostname is 127.0.0.1
   [junit4]   2> 7398 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Starting DataNode with maxLockedMemory = 0
   [junit4]   2> 7399 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Opened streaming server at /127.0.0.1:34571
   [junit4]   2> 7399 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Balancing bandwith is 1048576 bytes/s
   [junit4]   2> 7399 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Number threads for balancing is 5
   [junit4]   2> 7401 WARN  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7403 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
   [junit4]   2> 7411 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
   [junit4]   2> 7411 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
   [junit4]   2> 7411 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
   [junit4]   2> 7413 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.datanode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
   [junit4]   2> 7413 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.HttpServer2 Jetty bound to port 51484
   [junit4]   2> 7414 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7468 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51484_datanode____.uist8l/webapp
   [junit4]   2> 7774 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 7994 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage DataNode version: -56 and NameNode layout version: -60
   [junit4]   2> 7998 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 8001 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1 is not formatted for BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8001 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8008 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 8009 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2 is not formatted for BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8009 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8168 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Analyzing storage directories for bpid BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8168 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Locking is disabled
   [junit4]   2> 8170 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current/BP-1268983991-10.41.0.5-1449176489074 is not formatted.
   [junit4]   2> 8170 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8171 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting block pool BP-1268983991-10.41.0.5-1449176489074 directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current/BP-1268983991-10.41.0.5-1449176489074/current
   [junit4]   2> 8174 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Locking is disabled
   [junit4]   2> 8175 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current/BP-1268983991-10.41.0.5-1449176489074 is not formatted.
   [junit4]   2> 8175 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8175 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting block pool BP-1268983991-10.41.0.5-1449176489074 directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current/BP-1268983991-10.41.0.5-1449176489074/current
   [junit4]   2> 8178 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Restored 0 block files from trash.
   [junit4]   2> 8182 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51484
   [junit4]   2> 8185 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Restored 0 block files from trash.
   [junit4]   2> 8185 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode dnUserName = jenkins
   [junit4]   2> 8185 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode supergroup = supergroup
   [junit4]   2> 8185 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.i.CallQueueManager Using callQueue class java.util.concurrent.LinkedBlockingQueue
   [junit4]   2> 8187 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Setting up storage: nsid=1179921729;bpid=BP-1268983991-10.41.0.5-1449176489074;lv=-56;nsInfo=lv=-60;cid=testClusterID;nsid=1179921729;c=0;bpid=BP-1268983991-10.41.0.5-1449176489074;dnuuid=null
   [junit4]   2> 8190 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Generated and persisted new Datanode UUID a1a5894e-b5bd-430b-b6b0-af8a7bb588de
   [junit4]   2> 8191 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Opened IPC server at /127.0.0.1:49663
   [junit4]   2> 8200 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Refresh request received for nameservices: null
   [junit4]   2> 8201 INFO  (Socket Reader #1 for port 49663) [    ] o.a.h.i.Server Starting Socket Reader #1 for port 49663
   [junit4]   2> 8213 INFO  (SUITE-MorphlineBasicMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [    ] o.a.h.h.s.d.DataNode Starting BPOfferServices for nameservices: <default>
   [junit4]   2> 8217 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:43259 starting to offer service
   [junit4]   2> 8217 INFO  (IPC Server Responder) [    ] o.a.h.i.Server IPC Server Responder: starting
   [junit4]   2> 8218 INFO  (IPC Server listener on 49663) [    ] o.a.h.i.Server IPC Server listener on 49663: starting
   [junit4]   2> 8230 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage DataNode version: -56 and NameNode layout version: -60
   [junit4]   2> 8234 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 8235 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3 is not formatted for BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8235 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8240 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Lock on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/in_use.lock acquired by nodename 32282@lucene1-us-west
   [junit4]   2> 8250 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4 is not formatted for BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8250 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8309 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Added new volume: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current
   [junit4]   2> 8341 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Added volume - /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current, StorageType: DISK
   [junit4]   2> 8342 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Added new volume: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current
   [junit4]   2> 8342 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Added volume - /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current, StorageType: DISK
   [junit4]   2> 8391 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Registered FSDatasetState MBean
   [junit4]   2> 8402 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DirectoryScanner Periodic Directory Tree Verification scan starting at 1449189806630 with interval 21600000
   [junit4]   2> 8403 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Adding block pool BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8403 INFO  (Thread-89) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Scanning block pool BP-1268983991-10.41.0.5-1449176489074 on volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current...
   [junit4]   2> 8404 INFO  (Thread-90) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Scanning block pool BP-1268983991-10.41.0.5-1449176489074 on volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current...
   [junit4]   2> 8516 INFO  (Thread-89) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Time taken to scan block pool BP-1268983991-10.41.0.5-1449176489074 on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/current: 106ms
   [junit4]   2> 8516 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Analyzing storage directories for bpid BP-1268983991-10.41.0.5-1449176489074
   [junit4]   2> 8517 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Locking is disabled
   [junit4]   2> 8518 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/current/BP-1268983991-10.41.0.5-1449176489074 is not formatted.
   [junit4]   2> 8518 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8518 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting block pool BP-1268983991-10.41.0.5-1449176489074 directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/current/BP-1268983991-10.41.0.5-1449176489074/current
   [junit4]   2> 8521 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Locking is disabled
   [junit4]   2> 8522 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Storage directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/current/BP-1268983991-10.41.0.5-1449176489074 is not formatted.
   [junit4]   2> 8522 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting ...
   [junit4]   2> 8522 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Formatting block pool BP-1268983991-10.41.0.5-1449176489074 directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/current/BP-1268983991-10.41.0.5-1449176489074/current
   [junit4]   2> 8525 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Restored 0 block files from trash.
   [junit4]   2> 8525 INFO  (Thread-90) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Time taken to scan block pool BP-1268983991-10.41.0.5-1449176489074 on /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/current: 96ms
   [junit4]   2> 8525 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data2/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Total time to scan all replicas for block pool BP-1268983991-10.41.0.5-1449176489074: 122ms
   [junit4]   2> 8525 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.c.Storage Restored 0 block files from trash.
   [junit4]   2> 8537 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Setting up storage: nsid=1179921729;bpid=BP-1268983991-10.41.0.5-1449176489074;lv=-56;nsInfo=lv=-60;cid=testClusterID;nsid=1179921729;c=0;bpid=BP-1268983991-10.41.0.5-1449176489074;dnuuid=null
   [junit4]   2> 8541 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.DataNode Generated and persisted new Datanode UUID 59e613d4-abae-4102-9e3d-5075dd9f97bc
   [junit4]   2> 8542 INFO  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J0/temp/solr.hadoop.MorphlineBasicMiniMRTest_562707F3EC0A0F3D-001/tempDir-002/hdfs/build/dfs/data/data4/]]  heartbeating to localhost/127.0.0.1:43259) [    ] o.a.h.h.s.d.f.i.FsDatasetImpl Added new volume: /x1/jenkin

[...truncated too long message...]

r:core_node5 x:collection1] o.a.h.h.s.n.FSNamesystem Stopping services started for standby state
   [junit4]   2> 83287 INFO  (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 83389 INFO  (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.a.h.m.i.MetricsSystemImpl Stopping JobHistoryServer metrics system...
   [junit4]   2> 83391 ERROR (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.hadoop.MorphlineGoLiveMiniMRTest.teardownClass(MorphlineGoLiveMiniMRTest.java:236)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 53 more
   [junit4]   2> 83405 INFO  (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.a.h.m.i.MetricsSystemImpl JobHistoryServer metrics system stopped.
   [junit4]   2> 83405 INFO  (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.a.h.m.i.MetricsSystemImpl JobHistoryServer metrics system shutdown complete.
   [junit4]   2> 83406 INFO  (SUITE-MorphlineGoLiveMiniMRTest-seed#[562707F3EC0A0F3D]-worker) [n:127.0.0.1:47248_v_y c:collection1 s:shard1 r:core_node5 x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/contrib/solr-map-reduce/test/J1/temp/solr.hadoop.MorphlineGoLiveMiniMRTest_562707F3EC0A0F3D-001
   [junit4]   2> NOTE: test params are: codec=Lucene54, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sq_AL, timezone=Etc/GMT0
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=8,free=120967240,total=343408640
   [junit4]   2> NOTE: All tests run in this JVM: [MorphlineReducerTest, MapReduceIndexerToolArgumentParserTest, MorphlineGoLiveMiniMRTest]
   [junit4] Completed [7/7] on J1 in 76.59s, 2 tests, 1 error <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:799: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:736: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:253: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:467: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:526: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 7 suites, 35 tests, 2 errors, 1 ignored [seed: 562707F3EC0A0F3D]

Total time: 215 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1034 - Still Failing

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1034/

1 tests failed.
FAILED:  org.apache.lucene.search.TestGeoPointQuery.testRandomBig

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([6B493944E699B5AC:EC1E44CB77C0C92C]:0)
	at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:354)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:363)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131)
	at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:655)
	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:278)
	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:757)
	at org.apache.lucene.util.IOUtils.close(IOUtils.java:97)
	at org.apache.lucene.util.IOUtils.close(IOUtils.java:84)
	at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:766)
	at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
	at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)




Build Log:
[...truncated 8761 lines...]
   [junit4] Suite: org.apache.lucene.search.TestGeoPointQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestGeoPointQuery -Dtests.method=testRandomBig -Dtests.seed=6B493944E699B5AC -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ms -Dtests.timezone=Europe/Lisbon -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   2438s J1 | TestGeoPointQuery.testRandomBig <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6B493944E699B5AC:EC1E44CB77C0C92C]:0)
   [junit4]    > 	at org.apache.lucene.util.ArrayUtil.grow(ArrayUtil.java:354)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectField.<init>(DirectPostingsFormat.java:363)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat$DirectFields.<init>(DirectPostingsFormat.java:131)
   [junit4]    > 	at org.apache.lucene.codecs.memory.DirectPostingsFormat.fieldsProducer(DirectPostingsFormat.java:115)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:104)
   [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:65)
   [junit4]    > 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:655)
   [junit4]    > 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:278)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:757)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.close(IOUtils.java:97)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.close(IOUtils.java:84)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:766)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.doTestRandom(BaseGeoPointTestCase.java:399)
   [junit4]    > 	at org.apache.lucene.util.BaseGeoPointTestCase.testRandomBig(BaseGeoPointTestCase.java:327)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/sandbox/test/J1/temp/lucene.search.TestGeoPointQuery_6B493944E699B5AC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene54): {string=FST50, id=FST50, point=PostingsFormat(name=Direct)}, docValues:{id=DocValuesFormat(name=SimpleText), point=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=ms, timezone=Europe/Lisbon
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_80 (64-bit)/cpus=4,threads=1,free=331360376,total=398983168
   [junit4]   2> NOTE: All tests run in this JVM: [TestSlowFuzzyQuery2, TestGeoPointQuery]
   [junit4] Completed [19/20] on J1 in 2892.41s, 23 tests, 1 error <<< FAILURES!

[...truncated 4 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/sandbox/test/temp/junit4-J1-20151202_125238_127.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/heapdumps/java_pid29086.hprof ...
   [junit4] Heap dump file created [555886611 bytes in 4.516 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 10 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:792: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:729: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:471: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:2248: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1452: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1006: There were test failures: 20 suites, 142 tests, 1 error, 1 ignored (1 assumption) [seed: 6B493944E699B5AC]

Total time: 259 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any