You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/05/27 04:38:46 UTC
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 4 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/4/
5 tests failed.
REGRESSION: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=2882, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2882, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:47758: collection already exists: awholynewstresscollection_collection4_1
at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
REGRESSION: org.apache.solr.cloud.SyncSliceTest.test
Error Message:
timeout waiting to see all nodes active
Stack Trace:
java.lang.AssertionError: timeout waiting to see all nodes active
at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91:F4AFB4C31BB79269]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.SyncSliceTest.waitTillAllNodesActive(SyncSliceTest.java:256)
at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.TestMiniSolrCloudCluster.testBasics
Error Message:
expected:<5> but was:<4>
Stack Trace:
java.lang.AssertionError: expected:<5> but was:<4>
at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91:412325358DA5A1E1]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.TestMiniSolrCloudCluster.testCollectionCreateSearchDelete(TestMiniSolrCloudCluster.java:96)
at org.apache.solr.cloud.TestMiniSolrCloudCluster.testBasics(TestMiniSolrCloudCluster.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestMiniSolrCloudCluster
Error Message:
20 threads leaked from SUITE scope at org.apache.solr.cloud.TestMiniSolrCloudCluster: 1) Thread[id=60288, name=qtp2143160411-60288, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=60363, name=Thread-52030, state=WAITING, group=TGRP-TestMiniSolrCloudCluster] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1035) 3) Thread[id=60311, name=Scheduler-648987053, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=60378, name=OverseerCollectionProcessor-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer collection creation process.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342) at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:518) at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:276) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=60349, name=jetty-launcher-1913-thread-5-SendThread(127.0.0.1:51983), state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 6) Thread[id=60280, name=qtp2143160411-60280-acceptor-0@25b9f0e0-ServerConnector@2326130{HTTP/1.1}{127.0.0.1:43368}, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=60354, name=zkCallback-1930-thread-1, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=60314, name=org.eclipse.jetty.server.session.HashSessionManager@20128b27Timer, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=60271, name=qtp2143160411-60271-selector-ServerConnectorManager@7c6a4870/0, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=60300, name=qtp2143160411-60300, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 11) Thread[id=60379, name=OverseerHdfsCoreFailoverThread-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76) at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308) at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:772) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:152) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:131) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=60307, name=qtp2143160411-60307, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 13) Thread[id=60377, name=OverseerStateUpdate-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer state updater.] at java.lang.Object.wait(Native Method) at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:594) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:572) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:190) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=60386, name=zkCallback-1930-thread-4, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=60350, name=jetty-launcher-1913-thread-5-EventThread, state=WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 16) Thread[id=60306, name=qtp2143160411-60306, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=60376, name=zkCallback-1930-thread-2, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=60385, name=zkCallback-1930-thread-3, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=60277, name=qtp2143160411-60277-selector-ServerConnectorManager@7c6a4870/1, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 20) Thread[id=60295, name=qtp2143160411-60295, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.cloud.TestMiniSolrCloudCluster:
1) Thread[id=60288, name=qtp2143160411-60288, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=60363, name=Thread-52030, state=WAITING, group=TGRP-TestMiniSolrCloudCluster]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1035)
3) Thread[id=60311, name=Scheduler-648987053, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=60378, name=OverseerCollectionProcessor-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer collection creation process.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:518)
at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:276)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=60349, name=jetty-launcher-1913-thread-5-SendThread(127.0.0.1:51983), state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
6) Thread[id=60280, name=qtp2143160411-60280-acceptor-0@25b9f0e0-ServerConnector@2326130{HTTP/1.1}{127.0.0.1:43368}, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
7) Thread[id=60354, name=zkCallback-1930-thread-1, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=60314, name=org.eclipse.jetty.server.session.HashSessionManager@20128b27Timer, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
9) Thread[id=60271, name=qtp2143160411-60271-selector-ServerConnectorManager@7c6a4870/0, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
10) Thread[id=60300, name=qtp2143160411-60300, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
11) Thread[id=60379, name=OverseerHdfsCoreFailoverThread-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:108)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:76)
at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:772)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.doWork(OverseerAutoReplicaFailoverThread.java:152)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:131)
at java.lang.Thread.run(Thread.java:745)
12) Thread[id=60307, name=qtp2143160411-60307, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
13) Thread[id=60377, name=OverseerStateUpdate-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer state updater.]
at java.lang.Object.wait(Native Method)
at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276)
at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320)
at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:594)
at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:572)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:190)
at java.lang.Thread.run(Thread.java:745)
14) Thread[id=60386, name=zkCallback-1930-thread-4, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15) Thread[id=60350, name=jetty-launcher-1913-thread-5-EventThread, state=WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
16) Thread[id=60306, name=qtp2143160411-60306, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
17) Thread[id=60376, name=zkCallback-1930-thread-2, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
18) Thread[id=60385, name=zkCallback-1930-thread-3, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
19) Thread[id=60277, name=qtp2143160411-60277-selector-ServerConnectorManager@7c6a4870/1, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
20) Thread[id=60295, name=qtp2143160411-60295, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TestMiniSolrCloudCluster
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=60314, name=org.eclipse.jetty.server.session.HashSessionManager@20128b27Timer, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=60271, name=qtp2143160411-60271-selector-ServerConnectorManager@7c6a4870/0, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster] at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=60354, name=zkCallback-1930-thread-1, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=60311, name=Scheduler-648987053, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=60379, name=OverseerHdfsCoreFailoverThread-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139) at java.lang.Thread.run(Thread.java:745) 6) Thread[id=60363, name=Thread-52030, state=WAITING, group=TGRP-TestMiniSolrCloudCluster] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1035) 7) Thread[id=60386, name=zkCallback-1930-thread-4, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=60277, name=qtp2143160411-60277-selector-ServerConnectorManager@7c6a4870/1, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=60385, name=zkCallback-1930-thread-3, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=60349, name=jetty-launcher-1913-thread-5-SendThread(127.0.0.1:51983), state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 11) Thread[id=60376, name=zkCallback-1930-thread-2, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=60314, name=org.eclipse.jetty.server.session.HashSessionManager@20128b27Timer, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=60271, name=qtp2143160411-60271-selector-ServerConnectorManager@7c6a4870/0, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
at sun.nio.ch.IOUtil.drain(Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=60354, name=zkCallback-1930-thread-1, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=60311, name=Scheduler-648987053, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=60379, name=OverseerHdfsCoreFailoverThread-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
at java.lang.Thread.run(Thread.java:745)
6) Thread[id=60363, name=Thread-52030, state=WAITING, group=TGRP-TestMiniSolrCloudCluster]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1035)
7) Thread[id=60386, name=zkCallback-1930-thread-4, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=60277, name=qtp2143160411-60277-selector-ServerConnectorManager@7c6a4870/1, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
9) Thread[id=60385, name=zkCallback-1930-thread-3, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10) Thread[id=60349, name=jetty-launcher-1913-thread-5-SendThread(127.0.0.1:51983), state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
11) Thread[id=60376, name=zkCallback-1930-thread-2, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91]:0)
Build Log:
[...truncated 10451 lines...]
[junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/init-core-data-001
[junit4] 2> 308303 T1326 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 308303 T1326 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /jyl/h
[junit4] 2> 308311 T1326 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 308312 T1327 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 308312 T1327 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 308412 T1326 oasc.ZkTestServer.run start zk server on port:45021
[junit4] 2> 308412 T1326 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 308414 T1326 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 308417 T1334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b2a9a9 name:ZooKeeperConnection Watcher:127.0.0.1:45021 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 308417 T1326 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 308418 T1326 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 308418 T1326 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 308422 T1326 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 308427 T1326 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 308432 T1337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@351e4328 name:ZooKeeperConnection Watcher:127.0.0.1:45021/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 308432 T1326 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 308432 T1326 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 308433 T1326 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 308438 T1326 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 308442 T1326 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 308446 T1326 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 308451 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 308452 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 308459 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 308460 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 308462 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 308462 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 308464 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 308465 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 308468 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 308469 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 308472 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 308472 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 308477 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 308477 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 308482 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 308483 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 308486 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 308487 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 308489 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 308490 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 308493 T1326 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 308494 T1326 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 308861 T1326 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1
[junit4] 2> 308863 T1326 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 308866 T1326 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@45748af5{/jyl/h,null,AVAILABLE}
[junit4] 2> 308868 T1326 oejs.AbstractConnector.doStart Started ServerConnector@664efe72{HTTP/1.1}{127.0.0.1:33647}
[junit4] 2> 308869 T1326 oejs.Server.doStart Started @310398ms
[junit4] 2> 308869 T1326 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=33647, hostContext=/jyl/h, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/tempDir-001/control/data}
[junit4] 2> 308870 T1326 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@7b3cb2c6
[junit4] 2> 308871 T1326 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/'
[junit4] 2> 308899 T1326 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/solr.xml
[junit4] 2> 308921 T1326 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores
[junit4] 2> 308922 T1326 oasc.CoreContainer.<init> New CoreContainer 631396410
[junit4] 2> 308922 T1326 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/]
[junit4] 2> 308923 T1326 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/lib
[junit4] 2> 308923 T1326 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/lib).
[junit4] 2> 308933 T1326 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 308935 T1326 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 308935 T1326 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 308936 T1326 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 308936 T1326 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 308936 T1326 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45021/solr
[junit4] 2> 308936 T1326 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 308937 T1326 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 308937 T1326 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 308941 T1351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47351ad name:ZooKeeperConnection Watcher:127.0.0.1:45021 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 308943 T1326 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 308944 T1326 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 308947 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 308949 T1354 n:127.0.0.1:33647_jyl%2Fh oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5082c8 name:ZooKeeperConnection Watcher:127.0.0.1:45021/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 308949 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 308954 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 308958 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 308963 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 308967 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 308972 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 308978 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 308980 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 308982 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 308983 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33647_jyl%2Fh
[junit4] 2> 308984 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33647_jyl%2Fh
[junit4] 2> 308987 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 308990 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 308992 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 308993 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33647_jyl%2Fh
[junit4] 2> 308994 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 308997 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.Overseer.start Overseer (id=93892830167826435-127.0.0.1:33647_jyl%2Fh-n_0000000000) starting
[junit4] 2> 309000 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 309011 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 309012 T1326 n:127.0.0.1:33647_jyl%2Fh oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 309012 T1356 n:127.0.0.1:33647_jyl%2Fh oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 309013 T1355 n:127.0.0.1:33647_jyl%2Fh oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 309018 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 309019 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 309021 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores
[junit4] 2> 309022 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {loadOnStartup=true, dataDir=data/, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1, transient=false, name=collection1, coreNodeName=, shard=, schema=schema.xml, collection=control_collection, config=solrconfig.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/}
[junit4] 2> 309022 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/
[junit4] 2> 309023 T1326 n:127.0.0.1:33647_jyl%2Fh oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 309025 T1358 n:127.0.0.1:33647_jyl%2Fh c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 309026 T1358 n:127.0.0.1:33647_jyl%2Fh c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 309027 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 309029 T1355 n:127.0.0.1:33647_jyl%2Fh oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "node_name":"127.0.0.1:33647_jyl%2Fh",
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:33647/jyl/h",
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard":null} current state version: 0
[junit4] 2> 309031 T1355 n:127.0.0.1:33647_jyl%2Fh oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "node_name":"127.0.0.1:33647_jyl%2Fh",
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1:33647/jyl/h",
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard":null}
[junit4] 2> 309031 T1355 n:127.0.0.1:33647_jyl%2Fh oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 309031 T1355 n:127.0.0.1:33647_jyl%2Fh oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 309033 T1354 n:127.0.0.1:33647_jyl%2Fh oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 309038 T1354 n:127.0.0.1:33647_jyl%2Fh oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 310028 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 310028 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 310029 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 310029 T1358 n:127.0.0.1:33647_jyl%2Fh oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 310031 T1358 n:127.0.0.1:33647_jyl%2Fh oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 310031 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/'
[junit4] 2> 310054 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 310067 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 310099 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 310158 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 310160 T1358 n:127.0.0.1:33647_jyl%2Fh oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 310172 T1358 n:127.0.0.1:33647_jyl%2Fh oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 310560 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 310578 T1358 n:127.0.0.1:33647_jyl%2Fh oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 310581 T1358 n:127.0.0.1:33647_jyl%2Fh oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 310655 T1358 n:127.0.0.1:33647_jyl%2Fh oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 310661 T1358 n:127.0.0.1:33647_jyl%2Fh oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 310690 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 310693 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 310693 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 310693 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 310696 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 310697 T1358 n:127.0.0.1:33647_jyl%2Fh oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 310697 T1358 n:127.0.0.1:33647_jyl%2Fh oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 310697 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 310698 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 310698 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@72068d91
[junit4] 2> 310703 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/data
[junit4] 2> 310706 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/data/index/
[junit4] 2> 310706 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 310706 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J0/temp/solr.cloud.SyncSliceTest 7CFB8B19B54BFF91-001/control-001/cores/collection1/data/index
[junit4] 2> 310707 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=6, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=55.2802734375, floorSegmentMB=1.1806640625, forceMergeDeletesPctAllowed=4.765344884028098, segmentsPerTier=18.0, maxCFSSegmentSizeMB=2.02734375, noCFSRatio=0.6682638335091285
[junit4] 2> 310708 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@71477ff5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d21e7ed),segFN=segments_1,generation=1}
[junit4] 2> 310708 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 310715 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 310715 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 310716 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 310716 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 310716 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 310716 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 310717 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 310718 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 310718 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 310724 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 310728 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 310729 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 310731 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 310739 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: standard,/admin/properties,/replication,/admin/segments,/update/json,/schema,/admin/threads,/get,/config,/admin/luke,/update/json/docs,/admin/ping,/update,/admin/file,/admin/system,/admin/plugins,/admin/logging,/update/csv,/admin/mbeans
[junit4] 2> 310741 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 310742 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 310742 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 310743 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 310743 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 310744 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=1.2255859375, floorSegmentMB=1.890625, forceMergeDeletesPctAllowed=14.316198210102929, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 310745 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@71477ff5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d21e7ed),segFN=segments_1,generation=1}
[junit4] 2> 310745 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 310745 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@386b3d9a[collection1] main
[junit4] 2> 310747 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 310749 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 310749 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 310749 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 310749 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 310750 T1358 n:127.0.0.1:33647_jyl%2Fh x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/
[...truncated too long message...]
lectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=60354, name=zkCallback-1930-thread-1, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=60311, name=Scheduler-648987053, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 5) Thread[id=60379, name=OverseerHdfsCoreFailoverThread-93893022025777164-127.0.0.1:43368_solr-n_0000000003, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 6) Thread[id=60363, name=Thread-52030, state=WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:503)
[junit4] > at org.apache.solr.core.CloserThread.run(CoreContainer.java:1035)
[junit4] > 7) Thread[id=60386, name=zkCallback-1930-thread-4, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 8) Thread[id=60277, name=qtp2143160411-60277-selector-ServerConnectorManager@7c6a4870/1, state=RUNNABLE, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
[junit4] > at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
[junit4] > at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 9) Thread[id=60385, name=zkCallback-1930-thread-3, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 10) Thread[id=60349, name=jetty-launcher-1913-thread-5-SendThread(127.0.0.1:51983), state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 11) Thread[id=60376, name=zkCallback-1930-thread-2, state=TIMED_WAITING, group=TGRP-TestMiniSolrCloudCluster]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([7CFB8B19B54BFF91]:0)
[junit4] Completed [415/496] on J0 in 48.07s, 3 tests, 1 failure, 2 errors <<< FAILURES!
[...truncated 260 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1433: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 496 suites, 1973 tests, 2 suite-level errors, 1 error, 2 failures, 265 ignored (10 assumptions)
Total time: 176 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 944447253 bytes
Compression is 0.0%
Took 2 min 42 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 14 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/14/
1 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test
Error Message:
Error from server at http://127.0.0.1:60137/nobc/pn: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Error reading cluster properties Caused by: KeeperErrorCode = Session expired for /clusterprops.json
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:60137/nobc/pn: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Error reading cluster properties Caused by: KeeperErrorCode = Session expired for /clusterprops.json
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:325)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:442)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10778 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/init-core-data-001
[junit4] 2> 1019211 T5200 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 1019211 T5200 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nobc/pn
[junit4] 2> 1019216 T5200 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1019216 T5201 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1019216 T5201 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1019316 T5200 oasc.ZkTestServer.run start zk server on port:43356
[junit4] 2> 1019333 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1019335 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1019338 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1019339 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1019341 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1019343 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1019350 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1019352 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1019354 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1019355 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1019357 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1019364 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
[junit4] 2> 1019366 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
[junit4] 2> 1019368 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
[junit4] 2> 1019371 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1019372 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
[junit4] 2> 1019374 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
[junit4] 2> 1019376 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
[junit4] 2> 1019377 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
[junit4] 2> 1019379 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 1019380 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
[junit4] 2> 1019382 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
[junit4] 2> 1019384 T5200 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
[junit4] 2> 1019654 T5200 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1
[junit4] 2> 1019656 T5200 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1019658 T5200 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4ba8f27c{/nobc/pn,null,AVAILABLE}
[junit4] 2> 1019658 T5200 oejs.AbstractConnector.doStart Started ServerConnector@7e981e57{HTTP/1.1}{127.0.0.1:34468}
[junit4] 2> 1019659 T5200 oejs.Server.doStart Started @1021266ms
[junit4] 2> 1019659 T5200 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=34468, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores, hostContext=/nobc/pn}
[junit4] 2> 1019659 T5200 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 1019659 T5200 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/'
[junit4] 2> 1019673 T5200 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/solr.xml
[junit4] 2> 1019686 T5200 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores
[junit4] 2> 1019686 T5200 oasc.CoreContainer.<init> New CoreContainer 443947465
[junit4] 2> 1019686 T5200 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/]
[junit4] 2> 1019686 T5200 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/lib
[junit4] 2> 1019687 T5200 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/lib).
[junit4] 2> 1019694 T5200 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1019695 T5200 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1019695 T5200 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1019696 T5200 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1019696 T5200 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1019696 T5200 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43356/solr
[junit4] 2> 1019696 T5200 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1019713 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34468_nobc%2Fpn
[junit4] 2> 1019716 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1019717 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34468_nobc%2Fpn
[junit4] 2> 1019719 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer.start Overseer (id=93977739270291461-127.0.0.1:34468_nobc%2Fpn-n_0000000000) starting
[junit4] 2> 1019725 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1019726 T5236 n:127.0.0.1:34468_nobc%2Fpn oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1019732 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1019734 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1019735 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1019736 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores
[junit4] 2> 1019737 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {config=solrconfig.xml, dataDir=data/, shard=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1, name=collection1, loadOnStartup=true, transient=false, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/, coreNodeName=, schema=schema.xml, collection=control_collection}
[junit4] 2> 1019737 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/
[junit4] 2> 1019737 T5200 n:127.0.0.1:34468_nobc%2Fpn oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1019738 T5238 n:127.0.0.1:34468_nobc%2Fpn c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1019738 T5238 n:127.0.0.1:34468_nobc%2Fpn c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1019739 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1019739 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1019740 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:34468_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:34468/nobc/pn"} current state version: 0
[junit4] 2> 1019740 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:34468_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:34468/nobc/pn"}
[junit4] 2> 1019741 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 1019741 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1020739 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1020740 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1020740 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1020741 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/'
[junit4] 2> 1020755 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1020761 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1020773 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.1
[junit4] 2> 1020799 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1020800 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1020806 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1021031 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1021038 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1021040 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1021048 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1021051 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1021053 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1021054 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1021054 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1021054 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1021055 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1021055 T5238 n:127.0.0.1:34468_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1021055 T5238 n:127.0.0.1:34468_nobc%2Fpn oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1021055 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1021056 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1021056 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe2cdb4
[junit4] 2> 1021056 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/data
[junit4] 2> 1021057 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/data/index/
[junit4] 2> 1021057 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1021058 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/control-001/cores/collection1/data/index
[junit4] 2> 1021058 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 1021061 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@302b3e2e),segFN=segments_1,generation=1}
[junit4] 2> 1021061 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1021065 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1021065 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1021066 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1021066 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1021066 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1021066 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1021066 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1021067 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1021067 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1021069 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1021071 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1021072 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1021073 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1021077 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/luke,/schema,/admin/ping,/admin/plugins,/update/json,/admin/properties,/update,/replication,/admin/file,/config,/update/csv,/admin/mbeans,/admin/segments,standard,/admin/logging,/update/json/docs,/admin/system,/get,/admin/threads
[junit4] 2> 1021078 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1021078 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1021079 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1021079 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1021080 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1021080 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=85.091796875, floorSegmentMB=0.9384765625, forceMergeDeletesPctAllowed=23.308774821982688, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8488163824093705
[junit4] 2> 1021081 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@302b3e2e),segFN=segments_1,generation=1}
[junit4] 2> 1021081 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1021082 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@6c007057[collection1] main
[junit4] 2> 1021083 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1021083 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1021083 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1021083 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1021084 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1021084 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1021084 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1021084 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1021085 T5239 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6c007057[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1021085 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1021085 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1021085 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1021085 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503643830275014656
[junit4] 2> 1021087 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503643830275014656
[junit4] 2> 1021088 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1021088 T5238 n:127.0.0.1:34468_nobc%2Fpn x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1021090 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34468/nobc/pn collection:control_collection shard:shard1
[junit4] 2> 1021090 T5200 n:127.0.0.1:34468_nobc%2Fpn oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 1021090 T5200 n:127.0.0.1:34468_nobc%2Fpn oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1021096 T5200 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1021097 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1021098 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1021099 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1021099 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1021099 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C996 name=collection1 org.apache.solr.core.SolrCore@1ec2b59b url=http://127.0.0.1:34468/nobc/pn/collection1 node=127.0.0.1:34468_nobc%2Fpn C996_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:34468_nobc%2Fpn, core=collection1, state=down, base_url=http://127.0.0.1:34468/nobc/pn}
[junit4] 2> 1021099 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 C996 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34468/nobc/pn/collection1/
[junit4] 2> 1021100 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 C996 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1021100 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 C996 oasc.SyncStrategy.syncToMe http://127.0.0.1:34468/nobc/pn/collection1/ has no replicas
[junit4] 2> 1021100 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34468/nobc/pn/collection1/ shard1
[junit4] 2> 1021103 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1021104 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:34468/nobc/pn",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 1021255 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:34468/nobc/pn/collection1/ and leader is http://127.0.0.1:34468/nobc/pn/collection1/
[junit4] 2> 1021255 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34468/nobc/pn
[junit4] 2> 1021255 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1021256 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1021256 T5242 n:127.0.0.1:34468_nobc%2Fpn c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1021257 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1021259 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:34468_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "state":"active",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:34468/nobc/pn"} current state version: 2
[junit4] 2> 1021259 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:34468_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "state":"active",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:34468/nobc/pn"}
[junit4] 2> 1021412 T5200 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1
[junit4] 2> 1021413 T5200 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001
[junit4] 2> 1021425 T5200 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1021496 T5200 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@64fb3c37{/nobc/pn,null,AVAILABLE}
[junit4] 2> 1021496 T5200 oejs.AbstractConnector.doStart Started ServerConnector@4f6a8fd6{HTTP/1.1}{127.0.0.1:60137}
[junit4] 2> 1021497 T5200 oejs.Server.doStart Started @1023104ms
[junit4] 2> 1021497 T5200 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/nobc/pn, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores, solrconfig=solrconfig.xml, hostPort=60137}
[junit4] 2> 1021498 T5200 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 1021498 T5200 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/'
[junit4] 2> 1021512 T5200 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/solr.xml
[junit4] 2> 1021524 T5200 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores
[junit4] 2> 1021525 T5200 oasc.CoreContainer.<init> New CoreContainer 550665346
[junit4] 2> 1021525 T5200 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/]
[junit4] 2> 1021525 T5200 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/lib
[junit4] 2> 1021526 T5200 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/lib).
[junit4] 2> 1021533 T5200 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1021534 T5200 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1021535 T5200 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1021535 T5200 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1021535 T5200 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1021536 T5200 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43356/solr
[junit4] 2> 1021536 T5200 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1022548 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60137_nobc%2Fpn
[junit4] 2> 1022552 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1022554 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1022554 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1022555 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores
[junit4] 2> 1022556 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {shard=, config=solrconfig.xml, dataDir=data/, coreNodeName=, transient=false, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1, name=collection1, schema=schema.xml, collection=collection1, loadOnStartup=true, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/}
[junit4] 2> 1022556 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/
[junit4] 2> 1022556 T5200 n:127.0.0.1:60137_nobc%2Fpn oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1022557 T5263 n:127.0.0.1:60137_nobc%2Fpn c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1022558 T5263 n:127.0.0.1:60137_nobc%2Fpn c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1022559 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1022559 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1022560 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:60137_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60137/nobc/pn"} current state version: 3
[junit4] 2> 1022560 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:60137_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60137/nobc/pn"}
[junit4] 2> 1022561 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 1022561 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1023560 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1023560 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1023561 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1023561 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/'
[junit4] 2> 1023575 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1023580 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1023592 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.1
[junit4] 2> 1023630 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1023631 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1023636 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1023856 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1023863 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1023865 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1023873 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1023877 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1023879 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1023880 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1023880 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1023880 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1023881 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1023881 T5263 n:127.0.0.1:60137_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1023881 T5263 n:127.0.0.1:60137_nobc%2Fpn oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1023882 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1023882 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1023882 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe2cdb4
[junit4] 2> 1023883 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/data
[junit4] 2> 1023883 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 1023883 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1023884 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 1023884 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 1023887 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/index-SimpleFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@302b3e2e),segFN=segments_1,generation=1}
[junit4] 2> 1023887 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1023892 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1023892 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1023893 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1023893 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1023893 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1023893 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1023893 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1023894 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1023894 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1023897 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1023898 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1023899 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1023901 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1023906 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/luke,/schema,/admin/ping,/admin/plugins,/update/json,/admin/properties,/update,/replication,/admin/file,/config,/update/csv,/admin/mbeans,/admin/segments,standard,/admin/logging,/update/json/docs,/admin/system,/get,/admin/threads
[junit4] 2> 1023907 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1023908 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1023908 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1023909 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1023909 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1023910 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=85.091796875, floorSegmentMB=0.9384765625, forceMergeDeletesPctAllowed=23.308774821982688, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8488163824093705
[junit4] 2> 1023911 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/index-SimpleFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@302b3e2e),segFN=segments_1,generation=1}
[junit4] 2> 1023912 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1023912 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@92ccea6[collection1] main
[junit4] 2> 1023913 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1023913 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1023913 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1023914 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1023914 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1023914 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1023914 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1023914 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1023915 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1023915 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1023915 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1023915 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503643833242484736
[junit4] 2> 1023916 T5264 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@92ccea6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1023917 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503643833242484736
[junit4] 2> 1023917 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1023918 T5263 n:127.0.0.1:60137_nobc%2Fpn x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1023919 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60137/nobc/pn collection:collection1 shard:shard2
[junit4] 2> 1023919 T5200 n:127.0.0.1:60137_nobc%2Fpn oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 1023919 T5200 n:127.0.0.1:60137_nobc%2Fpn oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1023929 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1023930 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1023930 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1023931 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> 1023931 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C997 name=collection1 org.apache.solr.core.SolrCore@5a00e47b url=http://127.0.0.1:60137/nobc/pn/collection1 node=127.0.0.1:60137_nobc%2Fpn C997_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:60137_nobc%2Fpn, core=collection1, state=down, base_url=http://127.0.0.1:60137/nobc/pn}
[junit4] 2> 1023931 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 C997 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60137/nobc/pn/collection1/
[junit4] 2> 1023932 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 C997 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1023932 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 C997 oasc.SyncStrategy.syncToMe http://127.0.0.1:60137/nobc/pn/collection1/ has no replicas
[junit4] 2> 1023933 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60137/nobc/pn/collection1/ shard2
[junit4] 2> 1023936 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1023936 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60137/nobc/pn",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 1024087 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:60137/nobc/pn/collection1/ and leader is http://127.0.0.1:60137/nobc/pn/collection1/
[junit4] 2> 1024087 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60137/nobc/pn
[junit4] 2> 1024087 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1024088 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1024088 T5267 n:127.0.0.1:60137_nobc%2Fpn c:collection1 s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1024089 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1024090 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:60137_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "state":"active",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60137/nobc/pn"} current state version: 5
[junit4] 2> 1024090 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:60137_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "state":"active",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60137/nobc/pn"}
[junit4] 2> 1024214 T5200 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1
[junit4] 2> 1024215 T5200 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001
[junit4] 2> 1024216 T5200 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1024218 T5200 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@271b2ae{/nobc/pn,null,AVAILABLE}
[junit4] 2> 1024219 T5200 oejs.AbstractConnector.doStart Started ServerConnector@4d4aef5a{HTTP/1.1}{127.0.0.1:52852}
[junit4] 2> 1024219 T5200 oejs.Server.doStart Started @1025826ms
[junit4] 2> 1024219 T5200 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/nobc/pn, hostPort=52852, solrconfig=solrconfig.xml, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores}
[junit4] 2> 1024220 T5200 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 1024220 T5200 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/'
[junit4] 2> 1024234 T5200 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/solr.xml
[junit4] 2> 1024247 T5200 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores
[junit4] 2> 1024247 T5200 oasc.CoreContainer.<init> New CoreContainer 1075860511
[junit4] 2> 1024247 T5200 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/]
[junit4] 2> 1024247 T5200 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/lib
[junit4] 2> 1024248 T5200 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/lib).
[junit4] 2> 1024256 T5200 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1024257 T5200 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1024258 T5200 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1024258 T5200 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1024258 T5200 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1024258 T5200 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43356/solr
[junit4] 2> 1024259 T5200 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1025272 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52852_nobc%2Fpn
[junit4] 2> 1025276 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1025277 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1025277 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1025278 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores
[junit4] 2> 1025279 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {collection=collection1, coreNodeName=, name=collection1, loadOnStartup=true, shard=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1, transient=false, schema=schema.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/, config=solrconfig.xml, dataDir=data/}
[junit4] 2> 1025280 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/
[junit4] 2> 1025280 T5200 n:127.0.0.1:52852_nobc%2Fpn oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1025281 T5285 n:127.0.0.1:52852_nobc%2Fpn c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1025281 T5285 n:127.0.0.1:52852_nobc%2Fpn c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1025282 T5234 n:127.0.0.1:34468_nobc%2Fpn oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1025282 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1025283 T5235 n:127.0.0.1:34468_nobc%2Fpn oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:52852_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:52852/nobc/pn"} current state version: 6
[junit4] 2> 1025283 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "node_name":"127.0.0.1:52852_nobc%2Fpn",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "numShards":"2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:52852/nobc/pn"}
[junit4] 2> 1025284 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 1025284 T5235 n:127.0.0.1:34468_nobc%2Fpn oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1026283 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1026283 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1026284 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1026284 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/'
[junit4] 2> 1026298 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1026303 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1026315 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.1
[junit4] 2> 1026341 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1026342 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1026348 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1026564 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1026573 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1026574 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1026583 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1026589 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1026591 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1026592 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1026592 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1026593 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1026593 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1026593 T5285 n:127.0.0.1:52852_nobc%2Fpn oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1026594 T5285 n:127.0.0.1:52852_nobc%2Fpn oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1026594 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1026594 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1026594 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fe2cdb4
[junit4] 2> 1026595 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/data
[junit4] 2> 1026595 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/data/index/
[junit4] 2> 1026595 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1026596 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/shard-2-001/cores/collection1/data/index
[junit4] 2> 1026596 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 1026600 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001/index-SimpleFSDirectory-003 lockFactory=org.apache.lucene.store.NativeFSLockFactory@302b3e2e),segFN=segments_1,generation=1}
[junit4] 2> 1026600 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1026605 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1026606 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1026606 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1026606 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1026606 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1026606 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1026607 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1026607 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1026607 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1026610 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1026611 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1026613 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1026614 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1026626 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/luke,/schema,/admin/ping,/admin/plugins,/update/json,/admin/properties,/update,/replication,/admin/file,/config,/update/csv,/admin/mbeans,/admin/segments,standard,/admin/logging,/update/json/docs,/admin/system,/get,/admin/threads
[junit4] 2> 1026628 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1026641 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1026642 T5285 n:127.0.0.1:52852_nobc%2Fpn x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH nu
[...truncated too long message...]
utedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55426/nobc/pn, http://127.0.0.1:41294/nobc/pn, http://127.0.0.1:52852/nobc/pn, http://127.0.0.1:34468/nobc/pn, http://127.0.0.1:60137/nobc/pn]
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
[junit4] > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:52852/nobc/pn
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 more
[junit4] > Caused by: java.net.SocketTimeoutException: Read timed out
[junit4] > at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:152)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:122)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
[junit4] > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
[junit4] > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
[junit4] > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
[junit4] > ... 11 more
[junit4] 2> 1996782 T5200 c:collection1 s:shard1 x:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_436F3A4DF90D4A39-001
[junit4] 2> 977576 T5199 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=MockRandom), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), text=FST50, other_tl1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_l_dv=PostingsFormat(name=MockRandom), range_facet_i_dv=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=PostingsFormat(name=Asserting)}, docValues:{timestamp=DocValuesFormat(name=Asserting), range_facet_i_dv=DocValuesFormat(name=Asserting), range_facet_l_dv=DocValuesFormat(name=Memory)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=nl, timezone=America/North_Dakota/Beulah
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=52456984,total=522715136
[junit4] 2> NOTE: All tests run in this JVM: [TestComponentsName, TestCoreContainer, MultiThreadedOCPTest, SimpleMLTQParserTest, TestDynamicFieldCollectionResource, DocumentAnalysisRequestHandlerTest, TestSearchPerf, ShowFileRequestHandlerTest, TestFieldTypeCollectionResource, SpellPossibilityIteratorTest, TestDynamicLoading, TestElisionMultitermQuery, CacheHeaderTest, TestSolrQueryParserResource, UpdateParamsTest, TestValueSourceCache, NoCacheHeaderTest, TestUniqueKeyFieldResource, TestCharFilters, TestIntervalFaceting, CursorMarkTest, HdfsLockFactoryTest, TestStressVersions, TestHashQParserPlugin, SuggesterTSTTest, TestAtomicUpdateErrorCases, SolrCloudExampleTest, TestCollationFieldDocValues, TestMiniSolrCloudClusterSSL, EchoParamsTest, TestPivotHelperCode, TestHashPartitioner, TestPHPSerializedResponseWriter, TestShardHandlerFactory, TestOmitPositions, DocumentBuilderTest, JSONWriterTest, TestAnalyzedSuggestions, AsyncMigrateRouteKeyTest, DistributedMLTComponentTest, ZkCLITest, StressHdfsTest, TestNRTOpen, WordBreakSolrSpellCheckerTest, TestSolr4Spatial2, ConvertedLegacyTest, TestManagedResource, SolrTestCaseJ4Test, TestStandardQParsers, TestSchemaVersionResource, TestLMDirichletSimilarityFactory, AlternateDirectoryTest, PreAnalyzedFieldTest, DistanceFunctionTest, TriLevelCompositeIdRoutingTest, TestRequestStatusCollectionAPI, RAMDirectoryFactoryTest, ResourceLoaderTest, ExitableDirectoryReaderTest, DistribDocExpirationUpdateProcessorTest, DistributedFacetPivotLargeTest, TestBlobHandler, HdfsRecoveryZkTest, TestJettySolrRunner, ActionThrottleTest, TestLMJelinekMercerSimilarityFactory, DateRangeFieldTest, SolrPluginUtilsTest, TestLRUCache, SpatialFilterTest, StatelessScriptUpdateProcessorFactoryTest, LukeRequestHandlerTest, ChaosMonkeySafeLeaderTest, TestCloudInspectUtil, TestFuzzyAnalyzedSuggestions, TestRemoveLastDynamicCopyField, ResponseHeaderTest, TestClassNameShortening, TestWriterPerf, TestSchemaResource, CollectionsAPIAsyncDistributedZkTest, CollectionsAPIDistributedZkTest]
[junit4] Completed [259/498] on J1 in 978.10s, 1 test, 1 error <<< FAILURES!
[...truncated 764 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 498 suites, 1976 tests, 1 error, 52 ignored (10 assumptions)
Total time: 172 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 899930449 bytes
Compression is 0.0%
Took 2 min 34 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 13 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/13/
1 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=5187, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5187, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
at __randomizedtesting.SeedInfo.seed([47EA7E51B7880699]:0)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:884)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:53722: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
... 5 more
Build Log:
[...truncated 21210 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/init-core-data-001
[junit4] 2> 869111 T4831 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 869112 T4831 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 869116 T4831 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 869116 T4832 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 869116 T4832 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 869216 T4831 oasc.ZkTestServer.run start zk server on port:46516
[junit4] 2> 869217 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869217 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869220 T4839 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@294b5be8 name:ZooKeeperConnection Watcher:127.0.0.1:46516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869220 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869220 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869220 T4831 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 869223 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869223 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869224 T4842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b4cfbfa name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869225 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869225 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869225 T4831 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 869227 T4831 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 869229 T4831 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 869230 T4831 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 869232 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 869232 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 869235 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 869236 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 869238 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 869238 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 869240 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 869240 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 869242 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 869242 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 869244 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 869244 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 869247 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 869248 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 869249 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 869249 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 869251 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 869251 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 869253 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 869253 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 869254 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 869255 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 869257 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869257 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869259 T4845 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b6154f2 name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869259 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869259 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869261 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869261 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869262 T4848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a92590e name:ZooKeeperConnection Watcher:127.0.0.1:46516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869262 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869263 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869263 T4831 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 869264 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869265 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869266 T4851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bc1d5e7 name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869266 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869266 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869267 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
[junit4] 2> 869267 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
[junit4] 2> 869270 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
[junit4] 2> 869270 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
[junit4] 2> 869272 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
[junit4] 2> 869272 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/enumsConfig.xml
[junit4] 2> 869273 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 869274 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 869275 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
[junit4] 2> 869275 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
[junit4] 2> 869277 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
[junit4] 2> 869277 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
[junit4] 2> 869279 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
[junit4] 2> 869279 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
[junit4] 2> 869283 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
[junit4] 2> 869283 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
[junit4] 2> 869289 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 869289 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 869291 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
[junit4] 2> 869291 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
[junit4] 2> 869293 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
[junit4] 2> 869293 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
[junit4] 2> 869295 T4831 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
[junit4] 2> 869295 T4831 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
[junit4] 2> 869564 T4831 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1
[junit4] 2> 869565 T4831 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 869567 T4831 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@71f3387a{/,null,AVAILABLE}
[junit4] 2> 869579 T4831 oejs.AbstractConnector.doStart Started ServerConnector@45bced04{SSL-http/1.1}{127.0.0.1:53722}
[junit4] 2> 869579 T4831 oejs.Server.doStart Started @871048ms
[junit4] 2> 869580 T4831 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=53722, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores, hostContext=/}
[junit4] 2> 869581 T4831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 869581 T4831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/'
[junit4] 2> 869595 T4831 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/solr.xml
[junit4] 2> 869608 T4831 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores
[junit4] 2> 869608 T4831 oasc.CoreContainer.<init> New CoreContainer 1212142517
[junit4] 2> 869609 T4831 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/]
[junit4] 2> 869609 T4831 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/lib
[junit4] 2> 869609 T4831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/lib).
[junit4] 2> 869616 T4831 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 869618 T4831 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 869620 T4831 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 869620 T4831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 869620 T4831 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 869620 T4831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46516/solr
[junit4] 2> 869620 T4831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 869621 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 869621 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869622 T4865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14fbfc3 name:ZooKeeperConnection Watcher:127.0.0.1:46516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869623 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869623 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 869625 T4831 n:127.0.0.1:53722_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 869626 T4868 n:127.0.0.1:53722_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53c2b6ae name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 869626 T4831 n:127.0.0.1:53722_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 869627 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 869629 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 869631 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 869632 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 869634 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 869636 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 869638 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 869639 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 869640 T4831 n:127.0.0.1:53722_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53722_
[junit4] 2> 869641 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53722_
[junit4] 2> 869642 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 869643 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 869644 T4831 n:127.0.0.1:53722_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 869646 T4831 n:127.0.0.1:53722_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53722_
[junit4] 2> 869646 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 869647 T4831 n:127.0.0.1:53722_ oasc.Overseer.start Overseer (id=93971528274083846-127.0.0.1:53722_-n_0000000000) starting
[junit4] 2> 869649 T4831 n:127.0.0.1:53722_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 869653 T4831 n:127.0.0.1:53722_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 869654 T4870 n:127.0.0.1:53722_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 869654 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 869659 T4831 n:127.0.0.1:53722_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 869661 T4831 n:127.0.0.1:53722_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 869661 T4831 n:127.0.0.1:53722_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 869662 T4831 n:127.0.0.1:53722_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores
[junit4] 2> 869664 T4831 n:127.0.0.1:53722_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {transient=false, loadOnStartup=true, config=solrconfig.xml, collection=control_collection, dataDir=data/, name=collection1, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1, coreNodeName=, shard=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/, schema=schema.xml}
[junit4] 2> 869664 T4831 n:127.0.0.1:53722_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/
[junit4] 2> 869664 T4831 n:127.0.0.1:53722_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 869684 T4872 n:127.0.0.1:53722_ c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 869684 T4872 n:127.0.0.1:53722_ c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 869685 T4872 n:127.0.0.1:53722_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 869685 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 869686 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:53722",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:53722_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 869686 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:53722",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:53722_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"}
[junit4] 2> 869687 T4869 n:127.0.0.1:53722_ oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 869687 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 869688 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 870685 T4872 n:127.0.0.1:53722_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 870685 T4872 n:127.0.0.1:53722_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 870686 T4872 n:127.0.0.1:53722_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 870686 T4872 n:127.0.0.1:53722_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 870687 T4872 n:127.0.0.1:53722_ oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 870687 T4872 n:127.0.0.1:53722_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/'
[junit4] 2> 870700 T4872 n:127.0.0.1:53722_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 870706 T4872 n:127.0.0.1:53722_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 870718 T4872 n:127.0.0.1:53722_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 870744 T4872 n:127.0.0.1:53722_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 870745 T4872 n:127.0.0.1:53722_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 870753 T4872 n:127.0.0.1:53722_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 870995 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 871002 T4872 n:127.0.0.1:53722_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 871004 T4872 n:127.0.0.1:53722_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 871012 T4872 n:127.0.0.1:53722_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 871015 T4872 n:127.0.0.1:53722_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 871017 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 871018 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 871018 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 871018 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 871019 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 871019 T4872 n:127.0.0.1:53722_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 871019 T4872 n:127.0.0.1:53722_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 871019 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 871020 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 871020 T4872 n:127.0.0.1:53722_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4006c95c
[junit4] 2> 871021 T4872 n:127.0.0.1:53722_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/data
[junit4] 2> 871021 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/data/index/
[junit4] 2> 871021 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 871021 T4872 n:127.0.0.1:53722_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/control-001/cores/collection1/data/index
[junit4] 2> 871022 T4872 n:127.0.0.1:53722_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.748046875, noCFSRatio=0.893979828542094]
[junit4] 2> 871022 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7bb83c13 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7c7042d9),segFN=segments_1,generation=1}
[junit4] 2> 871022 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 871026 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 871027 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 871027 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 871027 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 871027 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 871027 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 871028 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 871028 T4872 n:127.0.0.1:53722_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 871028 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 871031 T4872 n:127.0.0.1:53722_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 871032 T4872 n:127.0.0.1:53722_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 871033 T4872 n:127.0.0.1:53722_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 871034 T4872 n:127.0.0.1:53722_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 871038 T4872 n:127.0.0.1:53722_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /config,/admin/system,/update/json/docs,/schema,/update,/admin/properties,/update/csv,/admin/luke,/admin/segments,/admin/file,/admin/ping,/admin/mbeans,/update/json,/admin/threads,/admin/logging,standard,/admin/plugins,/get,/replication
[junit4] 2> 871039 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 871039 T4872 n:127.0.0.1:53722_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 871040 T4872 n:127.0.0.1:53722_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 871040 T4872 n:127.0.0.1:53722_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 871041 T4872 n:127.0.0.1:53722_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 871041 T4872 n:127.0.0.1:53722_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=84.224609375, floorSegmentMB=1.7900390625, forceMergeDeletesPctAllowed=0.4324080580939038, segmentsPerTier=28.0, maxCFSSegmentSizeMB=1.0947265625, noCFSRatio=1.0
[junit4] 2> 871042 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7bb83c13 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7c7042d9),segFN=segments_1,generation=1}
[junit4] 2> 871042 T4872 n:127.0.0.1:53722_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 871042 T4872 n:127.0.0.1:53722_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@3c952835[collection1] main
[junit4] 2> 871042 T4872 n:127.0.0.1:53722_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 871043 T4872 n:127.0.0.1:53722_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 871043 T4872 n:127.0.0.1:53722_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 871044 T4872 n:127.0.0.1:53722_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 871044 T4872 n:127.0.0.1:53722_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 871044 T4872 n:127.0.0.1:53722_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 871044 T4872 n:127.0.0.1:53722_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 871044 T4872 n:127.0.0.1:53722_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 871045 T4872 n:127.0.0.1:53722_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 871045 T4872 n:127.0.0.1:53722_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 871045 T4873 n:127.0.0.1:53722_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c952835[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 871046 T4872 n:127.0.0.1:53722_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 871046 T4872 n:127.0.0.1:53722_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 871046 T4872 n:127.0.0.1:53722_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 871046 T4872 n:127.0.0.1:53722_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503544454406995968
[junit4] 2> 871048 T4872 n:127.0.0.1:53722_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503544454406995968
[junit4] 2> 871048 T4872 n:127.0.0.1:53722_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 871048 T4872 n:127.0.0.1:53722_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 871049 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53722 collection:control_collection shard:shard1
[junit4] 2> 871049 T4831 n:127.0.0.1:53722_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2
[junit4] 2> 871050 T4831 n:127.0.0.1:53722_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 871050 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 871053 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 871054 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 871055 T4879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c14d92e name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 871055 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 871055 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 871055 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 871056 T4831 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 871056 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 871057 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 871057 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 871057 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C644 name=collection1 org.apache.solr.core.SolrCore@626f1823 url=https://127.0.0.1:53722/collection1 node=127.0.0.1:53722_ C644_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:53722, node_name=127.0.0.1:53722_, state=down}
[junit4] 2> 871058 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 C644 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53722/collection1/
[junit4] 2> 871058 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 C644 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 871058 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 C644 oasc.SyncStrategy.syncToMe https://127.0.0.1:53722/collection1/ has no replicas
[junit4] 2> 871058 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53722/collection1/ shard1
[junit4] 2> 871058 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 871058 T4831 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 871061 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 871062 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"https://127.0.0.1:53722",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 871164 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871165 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871213 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are https://127.0.0.1:53722/collection1/ and leader is https://127.0.0.1:53722/collection1/
[junit4] 2> 871213 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53722
[junit4] 2> 871214 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 871214 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 871214 T4876 n:127.0.0.1:53722_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 871215 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 871216 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:53722",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "node_name":"127.0.0.1:53722_",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 871216 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:53722",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "node_name":"127.0.0.1:53722_",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state"}
[junit4] 2> 871319 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871319 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 871326 T4831 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1
[junit4] 2> 871327 T4831 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001
[junit4] 2> 871328 T4831 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 871332 T4831 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@f5cb48d{/,null,AVAILABLE}
[junit4] 2> 871336 T4831 oejs.AbstractConnector.doStart Started ServerConnector@3711375f{SSL-http/1.1}{127.0.0.1:60521}
[junit4] 2> 871336 T4831 oejs.Server.doStart Started @872805ms
[junit4] 2> 871337 T4831 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/, hostPort=60521, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores, solrconfig=solrconfig.xml}
[junit4] 2> 871338 T4831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 871338 T4831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/'
[junit4] 2> 871352 T4831 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/solr.xml
[junit4] 2> 871365 T4831 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores
[junit4] 2> 871365 T4831 oasc.CoreContainer.<init> New CoreContainer 375604993
[junit4] 2> 871366 T4831 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/]
[junit4] 2> 871366 T4831 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/lib
[junit4] 2> 871366 T4831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/lib).
[junit4] 2> 871373 T4831 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 871375 T4831 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 871377 T4831 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 871377 T4831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 871377 T4831 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 871377 T4831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46516/solr
[junit4] 2> 871377 T4831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 871378 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 871378 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 871387 T4893 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c5a5629 name:ZooKeeperConnection Watcher:127.0.0.1:46516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 871388 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 871388 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 871390 T4831 n:127.0.0.1:60521_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 871391 T4896 n:127.0.0.1:60521_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70801c3d name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 871392 T4831 n:127.0.0.1:60521_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 871396 T4831 n:127.0.0.1:60521_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 872399 T4831 n:127.0.0.1:60521_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60521_
[junit4] 2> 872400 T4831 n:127.0.0.1:60521_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60521_
[junit4] 2> 872402 T4831 n:127.0.0.1:60521_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 872403 T4831 n:127.0.0.1:60521_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 872404 T4831 n:127.0.0.1:60521_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 872404 T4831 n:127.0.0.1:60521_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores
[junit4] 2> 872405 T4831 n:127.0.0.1:60521_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {transient=false, collection=collection1, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1, config=solrconfig.xml, dataDir=data/, shard=, loadOnStartup=true, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/, schema=schema.xml, name=collection1, coreNodeName=}
[junit4] 2> 872405 T4831 n:127.0.0.1:60521_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/
[junit4] 2> 872406 T4831 n:127.0.0.1:60521_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 872407 T4897 n:127.0.0.1:60521_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 872407 T4897 n:127.0.0.1:60521_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 872408 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 872408 T4897 n:127.0.0.1:60521_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 872408 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:60521",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:60521_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 872410 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:60521",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:60521_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"}
[junit4] 2> 872410 T4869 n:127.0.0.1:53722_ oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 872411 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 872514 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 872514 T4896 n:127.0.0.1:60521_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 872514 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 873408 T4897 n:127.0.0.1:60521_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 873408 T4897 n:127.0.0.1:60521_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 873409 T4897 n:127.0.0.1:60521_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 873409 T4897 n:127.0.0.1:60521_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 873410 T4897 n:127.0.0.1:60521_ oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 873410 T4897 n:127.0.0.1:60521_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/'
[junit4] 2> 873423 T4897 n:127.0.0.1:60521_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 873428 T4897 n:127.0.0.1:60521_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 873440 T4897 n:127.0.0.1:60521_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 873466 T4897 n:127.0.0.1:60521_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 873467 T4897 n:127.0.0.1:60521_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 873473 T4897 n:127.0.0.1:60521_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 873714 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 873722 T4897 n:127.0.0.1:60521_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 873724 T4897 n:127.0.0.1:60521_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 873732 T4897 n:127.0.0.1:60521_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 873734 T4897 n:127.0.0.1:60521_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 873736 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 873737 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 873737 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 873737 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 873738 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 873738 T4897 n:127.0.0.1:60521_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 873738 T4897 n:127.0.0.1:60521_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 873738 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 873739 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 873739 T4897 n:127.0.0.1:60521_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4006c95c
[junit4] 2> 873740 T4897 n:127.0.0.1:60521_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/data
[junit4] 2> 873740 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 873740 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 873741 T4897 n:127.0.0.1:60521_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 873741 T4897 n:127.0.0.1:60521_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.748046875, noCFSRatio=0.893979828542094]
[junit4] 2> 873741 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@6ce66a3b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b487f58),segFN=segments_1,generation=1}
[junit4] 2> 873742 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 873746 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 873747 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 873747 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 873747 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 873747 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 873747 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 873748 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 873748 T4897 n:127.0.0.1:60521_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 873748 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 873750 T4897 n:127.0.0.1:60521_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 873752 T4897 n:127.0.0.1:60521_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 873753 T4897 n:127.0.0.1:60521_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 873754 T4897 n:127.0.0.1:60521_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 873759 T4897 n:127.0.0.1:60521_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /config,/admin/system,/update/json/docs,/schema,/update,/admin/properties,/update/csv,/admin/luke,/admin/segments,/admin/file,/admin/ping,/admin/mbeans,/update/json,/admin/threads,/admin/logging,standard,/admin/plugins,/get,/replication
[junit4] 2> 873760 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 873761 T4897 n:127.0.0.1:60521_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 873761 T4897 n:127.0.0.1:60521_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 873761 T4897 n:127.0.0.1:60521_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 873762 T4897 n:127.0.0.1:60521_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 873762 T4897 n:127.0.0.1:60521_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=84.224609375, floorSegmentMB=1.7900390625, forceMergeDeletesPctAllowed=0.4324080580939038, segmentsPerTier=28.0, maxCFSSegmentSizeMB=1.0947265625, noCFSRatio=1.0
[junit4] 2> 873763 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@6ce66a3b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b487f58),segFN=segments_1,generation=1}
[junit4] 2> 873763 T4897 n:127.0.0.1:60521_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 873763 T4897 n:127.0.0.1:60521_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@43aaf3b9[collection1] main
[junit4] 2> 873763 T4897 n:127.0.0.1:60521_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 873764 T4897 n:127.0.0.1:60521_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 873764 T4897 n:127.0.0.1:60521_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 873764 T4897 n:127.0.0.1:60521_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 873765 T4897 n:127.0.0.1:60521_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 873765 T4897 n:127.0.0.1:60521_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 873765 T4897 n:127.0.0.1:60521_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 873765 T4897 n:127.0.0.1:60521_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 873765 T4897 n:127.0.0.1:60521_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 873766 T4897 n:127.0.0.1:60521_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 873766 T4898 n:127.0.0.1:60521_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43aaf3b9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 873766 T4897 n:127.0.0.1:60521_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 873767 T4897 n:127.0.0.1:60521_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 873767 T4897 n:127.0.0.1:60521_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 873767 T4897 n:127.0.0.1:60521_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503544457260171264
[junit4] 2> 873769 T4897 n:127.0.0.1:60521_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503544457260171264
[junit4] 2> 873769 T4897 n:127.0.0.1:60521_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 873769 T4897 n:127.0.0.1:60521_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 873770 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60521 collection:collection1 shard:shard2
[junit4] 2> 873770 T4831 n:127.0.0.1:60521_ oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2
[junit4] 2> 873770 T4831 n:127.0.0.1:60521_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 873772 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 873777 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 873779 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 873779 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 873779 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C645 name=collection1 org.apache.solr.core.SolrCore@131ec282 url=https://127.0.0.1:60521/collection1 node=127.0.0.1:60521_ C645_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:60521, node_name=127.0.0.1:60521_, state=down}
[junit4] 2> 873779 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 C645 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60521/collection1/
[junit4] 2> 873779 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> 873779 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 C645 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 873780 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 C645 oasc.SyncStrategy.syncToMe https://127.0.0.1:60521/collection1/ has no replicas
[junit4] 2> 873780 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60521/collection1/ shard2
[junit4] 2> 873780 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 873783 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 873784 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"https://127.0.0.1:60521",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 873886 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 873886 T4896 n:127.0.0.1:60521_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 873886 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 873934 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register We are https://127.0.0.1:60521/collection1/ and leader is https://127.0.0.1:60521/collection1/
[junit4] 2> 873935 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60521
[junit4] 2> 873935 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 873935 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 873935 T4901 n:127.0.0.1:60521_ c:collection1 s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 873940 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 873941 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:60521",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "node_name":"127.0.0.1:60521_",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state"} current state version: 5
[junit4] 2> 873942 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:60521",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "node_name":"127.0.0.1:60521_",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state"}
[junit4] 2> 874044 T4879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 874044 T4868 n:127.0.0.1:53722_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 874044 T4896 n:127.0.0.1:60521_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 874045 T4831 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores/collection1
[junit4] 2> 874045 T4831 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001
[junit4] 2> 874046 T4831 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 874049 T4831 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@45a3ab96{/,null,AVAILABLE}
[junit4] 2> 874052 T4831 oejs.AbstractConnector.doStart Started ServerConnector@7c562891{SSL-http/1.1}{127.0.0.1:42767}
[junit4] 2> 874052 T4831 oejs.Server.doStart Started @875521ms
[junit4] 2> 874053 T4831 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=42767, hostContext=/, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores, solrconfig=solrconfig.xml}
[junit4] 2> 874054 T4831 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 874054 T4831 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/'
[junit4] 2> 874068 T4831 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/solr.xml
[junit4] 2> 874081 T4831 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores
[junit4] 2> 874081 T4831 oasc.CoreContainer.<init> New CoreContainer 1991570592
[junit4] 2> 874081 T4831 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/]
[junit4] 2> 874081 T4831 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/lib
[junit4] 2> 874082 T4831 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/lib).
[junit4] 2> 874089 T4831 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 874092 T4831 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 874093 T4831 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 874093 T4831 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 874094 T4831 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 874094 T4831 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46516/solr
[junit4] 2> 874094 T4831 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 874094 T4831 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 874096 T4831 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 874096 T4915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a13a81b name:ZooKeeperConnection Watcher:127.0.0.1:46516 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 874097 T4831 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 874097 T4831 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 874099 T4831 n:127.0.0.1:42767_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 874100 T4918 n:127.0.0.1:42767_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30aab969 name:ZooKeeperConnection Watcher:127.0.0.1:46516/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 874100 T4831 n:127.0.0.1:42767_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 874104 T4831 n:127.0.0.1:42767_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 875108 T4831 n:127.0.0.1:42767_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42767_
[junit4] 2> 875109 T4831 n:127.0.0.1:42767_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42767_
[junit4] 2> 875111 T4831 n:127.0.0.1:42767_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 875113 T4831 n:127.0.0.1:42767_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 875114 T4831 n:127.0.0.1:42767_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 875115 T4831 n:127.0.0.1:42767_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores
[junit4] 2> 875116 T4831 n:127.0.0.1:42767_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {schema=schema.xml, name=collection1, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores/collection1/, dataDir=data/, shard=, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores/collection1, config=solrconfig.xml, collection=collection1, coreNodeName=, loadOnStartup=true, transient=false}
[junit4] 2> 875116 T4831 n:127.0.0.1:42767_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_47EA7E51B7880699-001/shard-2-001/cores/collection1/
[junit4] 2> 875117 T4831 n:127.0.0.1:42767_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 875118 T4919 n:127.0.0.1:42767_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 875118 T4919 n:127.0.0.1:42767_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 875119 T4868 n:127.0.0.1:53722_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 875119 T4919 n:127.0.0.1:42767_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 875119 T4869 n:127.0.0.1:53722_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:42767",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:42767_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"} current state version: 6
[junit4] 2> 875120 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:42767",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:42767_",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state"}
[junit4] 2> 875120 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 875120 T4869 n:127.0.0.1:53722_ oasco.ReplicaMutator.updateState Assigning new node to shard
[...truncated too long message...]
SolrException: Error from server at https://127.0.0.1:53722: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 moreThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5185, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
[junit4] > at __randomizedtesting.SeedInfo.seed([47EA7E51B7880699]:0)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
[junit4] > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
[junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:53722: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 moreThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5184, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
[junit4] > at __randomizedtesting.SeedInfo.seed([47EA7E51B7880699]:0)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56121, https://127.0.0.1:57203, https://127.0.0.1:53722, https://127.0.0.1:60521, https://127.0.0.1:42767]
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
[junit4] > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
[junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:53722: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 more
[junit4] 2> 1443717 T4831 c:collection1 s:shard2 x:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=uk_UA, timezone=Antarctica/Troll
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=106908456,total=531103744
[junit4] 2> NOTE: All tests run in this JVM: [TestMissingGroups, SoftAutoCommitTest, TestRebalanceLeaders, SearchHandlerTest, TestReload, QueryResultKeyTest, TestSortingResponseWriter, OverseerTest, SynonymTokenizerTest, TestBM25SimilarityFactory, ResponseLogComponentTest, TestCollationField, TestQueryUtils, PolyFieldTest, TestFieldTypeResource, TestSolrConfigHandler, ZkStateWriterTest, TestDefaultSimilarityFactory, AnalysisErrorHandlingTest, TestFreeTextSuggestions, DOMUtilTest, TestLeaderElectionZkExpiry, TestCopyFieldCollectionResource, TestGroupingSearch, OutputWriterTest, TestBulkSchemaAPI, InfoHandlerTest, TestNoOpRegenerator, TestSimpleQParserPlugin, TestRecoveryHdfs, TestRecovery, SortByFunctionTest, LeaderElectionTest, DirectUpdateHandlerTest, RegexBoostProcessorTest, TestSimpleTrackingShardHandler, PrimUtilsTest, TestFieldSortValues, LoggingHandlerTest, SSLMigrationTest, SolrCoreCheckLockOnStartupTest, TestManagedSchemaFieldResource, CustomCollectionTest, SaslZkACLProviderTest, RemoteQueryErrorTest, OpenExchangeRatesOrgProviderTest, SampleTest, RecoveryZkTest, TestTrackingShardHandlerFactory, HardAutoCommitTest, XsltUpdateRequestHandlerTest, TestSort, TestUtils, AtomicUpdatesTest, TestFieldCollectionResource, TestCollectionAPI, QueryParsingTest, TestSerializedLuceneMatchVersion, CursorPagingTest, OverseerRolesTest, TestMacros, TestBulkSchemaConcurrent, TestDistribDocBasedVersion, OverseerCollectionProcessorTest, TestExtendedDismaxParser, SegmentsInfoRequestHandlerTest, DeleteReplicaTest, TestSolrQueryParserDefaultOperatorResource, TestConfigOverlay, TestXIncludeConfig, ReplicationFactorTest, DocValuesTest, TestInfoStreamLogging, QueryEqualityTest, BinaryUpdateRequestHandlerTest, CloudMLTQParserTest, BadIndexSchemaTest, TestSystemIdResolver, TestSchemaNameResource, CollectionsAPIDistributedZkTest]
[junit4] Completed [238/497] on J2 in 575.38s, 1 test, 1 error <<< FAILURES!
[...truncated 830 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 497 suites, 1975 tests, 1 error, 52 ignored (10 assumptions)
Total time: 170 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 850616672 bytes
Compression is 0.0%
Took 2 min 57 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 12 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/12/
No tests ran.
Build Log:
[...truncated 123 lines...]
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_2
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2 at hudson.remoting.Channel@1c3ac5fa:lucene
at hudson.FilePath.act(FilePath.java:916)
at hudson.FilePath.act(FilePath.java:893)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
at hudson.model.Run.execute(Run.java:1706)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_2 at revision '2015-06-05T18:52:14.344 -0400'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_2
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2 at hudson.remoting.Channel@1c3ac5fa:lucene
at hudson.FilePath.act(FilePath.java:916)
at hudson.FilePath.act(FilePath.java:893)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
at hudson.model.Run.execute(Run.java:1706)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_2 at revision '2015-06-05T18:52:14.344 -0400'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_2
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2 at hudson.remoting.Channel@1c3ac5fa:lucene
at hudson.FilePath.act(FilePath.java:916)
at hudson.FilePath.act(FilePath.java:893)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
at hudson.model.Run.execute(Run.java:1706)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_5_2 failed
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_5_2'
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
... 35 more
Caused by: svn: E175002: connection refused by the server
at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
... 35 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
... 35 more
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 0.19 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 11 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/11/
All tests passed
Build Log:
[...truncated 12090 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/init-core-data-001
[junit4] 2> 1343166 T10984 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 1343166 T10984 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /u_mbs/zf
[junit4] 2> 1343170 T10984 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1343171 T10985 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1343171 T10985 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1343271 T10984 oasc.ZkTestServer.run start zk server on port:46883
[junit4] 2> 1343271 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1343272 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343274 T10992 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70b7f642 name:ZooKeeperConnection Watcher:127.0.0.1:46883 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343275 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343275 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1343275 T10984 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1343278 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1343278 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343279 T10995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d7c7c82 name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343280 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343280 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1343280 T10984 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1343282 T10984 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1343284 T10984 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1343285 T10984 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1343287 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1343287 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1343290 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1343290 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1343292 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1343292 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1343294 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1343294 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1343296 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1343296 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1343298 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1343298 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1343300 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1343300 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1343302 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1343302 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1343304 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1343304 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1343306 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1343306 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1343307 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1343308 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1343310 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1343310 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343311 T10998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@585590d5 name:ZooKeeperConnection Watcher:127.0.0.1:46883 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343312 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343312 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1343312 T10984 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1343314 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1343314 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343315 T11001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bc24564 name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343315 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343316 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1343316 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
[junit4] 2> 1343317 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
[junit4] 2> 1343319 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
[junit4] 2> 1343319 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
[junit4] 2> 1343321 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
[junit4] 2> 1343321 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/enumsConfig.xml
[junit4] 2> 1343323 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1343323 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1343325 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
[junit4] 2> 1343325 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
[junit4] 2> 1343326 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
[junit4] 2> 1343327 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
[junit4] 2> 1343328 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
[junit4] 2> 1343329 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
[junit4] 2> 1343330 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
[junit4] 2> 1343330 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
[junit4] 2> 1343332 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 1343332 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 1343334 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
[junit4] 2> 1343334 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
[junit4] 2> 1343336 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
[junit4] 2> 1343336 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
[junit4] 2> 1343338 T10984 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
[junit4] 2> 1343338 T10984 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
[junit4] 2> 1343630 T10984 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1
[junit4] 2> 1343632 T10984 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1343633 T10984 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@53e240b9{/u_mbs/zf,null,AVAILABLE}
[junit4] 2> 1343637 T10984 oejs.AbstractConnector.doStart Started ServerConnector@70578524{HTTP/1.1}{127.0.0.1:43567}
[junit4] 2> 1343637 T10984 oejs.Server.doStart Started @1345072ms
[junit4] 2> 1343637 T10984 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores, hostPort=43567, hostContext=/u_mbs/zf}
[junit4] 2> 1343638 T10984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@7b3cb2c6
[junit4] 2> 1343638 T10984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/'
[junit4] 2> 1343656 T10984 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/solr.xml
[junit4] 2> 1343679 T10984 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores
[junit4] 2> 1343679 T10984 oasc.CoreContainer.<init> New CoreContainer 712310242
[junit4] 2> 1343680 T10984 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/]
[junit4] 2> 1343680 T10984 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/lib
[junit4] 2> 1343680 T10984 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/lib).
[junit4] 2> 1343688 T10984 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1343689 T10984 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1343690 T10984 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1343690 T10984 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1343690 T10984 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1343691 T10984 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46883/solr
[junit4] 2> 1343691 T10984 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1343691 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1343692 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343693 T11015 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c408dd2 name:ZooKeeperConnection Watcher:127.0.0.1:46883 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343694 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343695 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1343697 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1343698 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a1852c3 name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1343698 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1343700 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1343702 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1343705 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1343708 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1343710 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1343712 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1343714 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1343716 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1343717 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43567_u_mbs%2Fzf
[junit4] 2> 1343718 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43567_u_mbs%2Fzf
[junit4] 2> 1343720 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1343721 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1343723 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1343724 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43567_u_mbs%2Fzf
[junit4] 2> 1343724 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1343726 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer.start Overseer (id=93943302847528965-127.0.0.1:43567_u_mbs%2Fzf-n_0000000000) starting
[junit4] 2> 1343728 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1343735 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1343735 T11020 n:127.0.0.1:43567_u_mbs%2Fzf oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1343735 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1343736 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1343739 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1343740 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1343741 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores
[junit4] 2> 1343742 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1, collection=control_collection, schema=schema.xml, config=solrconfig.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/, shard=, dataDir=data/, transient=false, loadOnStartup=true, coreNodeName=, name=collection1}
[junit4] 2> 1343743 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/
[junit4] 2> 1343743 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1343744 T11022 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1343744 T11022 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1343745 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1343745 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1343746 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "base_url":"http://127.0.0.1:43567/u_mbs/zf",
[junit4] 2> "numShards":"1",
[junit4] 2> "node_name":"127.0.0.1:43567_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection"} current state version: 0
[junit4] 2> 1343746 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "base_url":"http://127.0.0.1:43567/u_mbs/zf",
[junit4] 2> "numShards":"1",
[junit4] 2> "node_name":"127.0.0.1:43567_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection"}
[junit4] 2> 1343746 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 1343747 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1343748 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1344745 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1344745 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1344746 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1344746 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1344747 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1344747 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/'
[junit4] 2> 1344762 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1344768 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1344782 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 1344809 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1344810 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1344817 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1345047 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1345054 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1345056 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1345065 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1345068 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1345070 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1345071 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1345071 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1345072 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1345072 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1345072 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1345073 T11022 n:127.0.0.1:43567_u_mbs%2Fzf oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1345073 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1345073 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1345073 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@158c156
[junit4] 2> 1345074 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/data
[junit4] 2> 1345075 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/data/index/
[junit4] 2> 1345075 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1345075 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/control-001/cores/collection1/data/index
[junit4] 2> 1345075 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.33203125, noCFSRatio=1.0]
[junit4] 2> 1345076 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@1a3978e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@332f093a; maxCacheMB=0.025390625 maxMergeSizeMB=0.826171875)),segFN=segments_1,generation=1}
[junit4] 2> 1345076 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1345080 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1345081 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1345081 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1345081 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1345081 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1345081 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1345082 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1345082 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1345082 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1345085 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1345086 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1345087 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1345089 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1345092 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/ping,/config,standard,/admin/mbeans,/admin/logging,/admin/luke,/admin/file,/schema,/update,/admin/plugins,/update/csv,/admin/segments,/admin/properties,/admin/threads,/admin/system,/update/json,/replication,/get,/update/json/docs
[junit4] 2> 1345093 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1345094 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1345094 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1345095 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1345095 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1345096 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=61.6533203125, floorSegmentMB=1.2470703125, forceMergeDeletesPctAllowed=4.5501836644801195, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6703114503366376
[junit4] 2> 1345096 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@1a3978e1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@332f093a; maxCacheMB=0.025390625 maxMergeSizeMB=0.826171875)),segFN=segments_1,generation=1}
[junit4] 2> 1345097 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1345097 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@26a8671c[collection1] main
[junit4] 2> 1345097 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1345098 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1345098 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1345099 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1345099 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1345099 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1345099 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1345099 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1345100 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1345100 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1345101 T11023 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@26a8671c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1345101 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1345101 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1345101 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1345101 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503092847582117888
[junit4] 2> 1345103 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503092847582117888
[junit4] 2> 1345103 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1345104 T11022 n:127.0.0.1:43567_u_mbs%2Fzf x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1345105 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43567/u_mbs/zf collection:control_collection shard:shard1
[junit4] 2> 1345105 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2
[junit4] 2> 1345105 T10984 n:127.0.0.1:43567_u_mbs%2Fzf oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1345106 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1345107 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1345107 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1345109 T11029 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48c65099 name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1345109 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1345109 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1345110 T10984 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1345110 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1345111 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1345112 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1345112 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2999 name=collection1 org.apache.solr.core.SolrCore@1ab95037 url=http://127.0.0.1:43567/u_mbs/zf/collection1 node=127.0.0.1:43567_u_mbs%2Fzf C2999_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:43567/u_mbs/zf, node_name=127.0.0.1:43567_u_mbs%2Fzf, core=collection1, state=down}
[junit4] 2> 1345112 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 C2999 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43567/u_mbs/zf/collection1/
[junit4] 2> 1345112 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> 1345112 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 C2999 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1345113 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 C2999 oasc.SyncStrategy.syncToMe http://127.0.0.1:43567/u_mbs/zf/collection1/ has no replicas
[junit4] 2> 1345113 T10984 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1345113 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43567/u_mbs/zf/collection1/ shard1
[junit4] 2> 1345113 T10984 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
[junit4] 2> 1345114 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1345114 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1345114 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1345116 T11032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a43082c name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1345116 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1345116 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1345118 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1345119 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
[junit4] 2> "operation":"create",
[junit4] 2> "name":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "stateFormat":"2"} current state version: 1
[junit4] 2> 1345119 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 1345119 T11029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345124 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345127 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:43567/u_mbs/zf",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 2
[junit4] 2> 1345128 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
[junit4] 2> 1345129 T11029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345129 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345132 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1345132 T11029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345133 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345169 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:43567/u_mbs/zf/collection1/ and leader is http://127.0.0.1:43567/u_mbs/zf/collection1/
[junit4] 2> 1345169 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43567/u_mbs/zf
[junit4] 2> 1345169 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1345169 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1345169 T11026 n:127.0.0.1:43567_u_mbs%2Fzf c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1345170 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1345171 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "base_url":"http://127.0.0.1:43567/u_mbs/zf",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:43567_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection"} current state version: 4
[junit4] 2> 1345172 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"http://127.0.0.1:43567/u_mbs/zf",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:43567_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection"}
[junit4] 2> 1345275 T11029 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345275 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1345426 T10984 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1
[junit4] 2> 1345427 T10984 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001
[junit4] 2> 1345428 T10984 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1345429 T10984 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7c574019{/u_mbs/zf,null,AVAILABLE}
[junit4] 2> 1345430 T10984 oejs.AbstractConnector.doStart Started ServerConnector@7ea04ad{HTTP/1.1}{127.0.0.1:36903}
[junit4] 2> 1345430 T10984 oejs.Server.doStart Started @1346865ms
[junit4] 2> 1345431 T10984 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solrconfig=solrconfig.xml, hostContext=/u_mbs/zf, hostPort=36903, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores}
[junit4] 2> 1345431 T10984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@7b3cb2c6
[junit4] 2> 1345431 T10984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/'
[junit4] 2> 1345446 T10984 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/solr.xml
[junit4] 2> 1345459 T10984 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores
[junit4] 2> 1345460 T10984 oasc.CoreContainer.<init> New CoreContainer 538700096
[junit4] 2> 1345460 T10984 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/]
[junit4] 2> 1345460 T10984 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/lib
[junit4] 2> 1345461 T10984 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/lib).
[junit4] 2> 1345468 T10984 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1345470 T10984 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1345470 T10984 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1345470 T10984 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1345471 T10984 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1345471 T10984 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46883/solr
[junit4] 2> 1345471 T10984 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1345471 T10984 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1345472 T10984 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1345474 T11046 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55cb72e3 name:ZooKeeperConnection Watcher:127.0.0.1:46883 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1345474 T10984 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1345475 T10984 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1345476 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1345477 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f9ef007 name:ZooKeeperConnection Watcher:127.0.0.1:46883/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1345478 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1345482 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1346488 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36903_u_mbs%2Fzf
[junit4] 2> 1346489 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36903_u_mbs%2Fzf
[junit4] 2> 1346492 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1346494 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1346495 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1346495 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores
[junit4] 2> 1346496 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {dataDir=data/, name=collection1, schema=schema.xml, shard=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/, transient=false, collection=collection1, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1, coreNodeName=, config=solrconfig.xml}
[junit4] 2> 1346497 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/
[junit4] 2> 1346497 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1346500 T11050 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1346500 T11050 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1346501 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1346502 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.preRegister Registering watch for external collection collection1
[junit4] 2> 1346502 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.addZkWatch addZkWatch collection1
[junit4] 2> 1346502 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "base_url":"http://127.0.0.1:36903/u_mbs/zf",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:36903_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1"} current state version: 5
[junit4] 2> 1346502 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper...
[junit4] 2> 1346503 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"http://127.0.0.1:36903/u_mbs/zf",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:36903_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1"}
[junit4] 2> 1346503 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 1346503 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1346504 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0
[junit4] 2> 1346504 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1346606 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
[junit4] 2> 1346607 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
[junit4] 2> 1346608 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1
[junit4] 2> 1347504 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1347505 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1347505 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1347505 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1347506 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1347506 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/'
[junit4] 2> 1347523 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1347530 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1347544 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 1347573 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1347574 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1347582 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1347854 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1347862 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1347864 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1347874 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1347877 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1347880 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1347881 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1347881 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1347881 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1347882 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1347882 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1347882 T11050 n:127.0.0.1:36903_u_mbs%2Fzf oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1347882 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1347883 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1347883 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@158c156
[junit4] 2> 1347884 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/data
[junit4] 2> 1347884 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 1347885 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1347885 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 1347885 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.33203125, noCFSRatio=1.0]
[junit4] 2> 1347886 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@39413843 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@68d4d412; maxCacheMB=0.025390625 maxMergeSizeMB=0.826171875)),segFN=segments_1,generation=1}
[junit4] 2> 1347886 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1347891 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1347891 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1347891 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1347892 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1347892 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1347892 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1347892 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1347893 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1347893 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1347895 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1347897 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1347898 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1347901 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1347908 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/ping,/config,standard,/admin/mbeans,/admin/logging,/admin/luke,/admin/file,/schema,/update,/admin/plugins,/update/csv,/admin/segments,/admin/properties,/admin/threads,/admin/system,/update/json,/replication,/get,/update/json/docs
[junit4] 2> 1347909 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1347910 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1347910 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1347911 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1347911 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1347913 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=61.6533203125, floorSegmentMB=1.2470703125, forceMergeDeletesPctAllowed=4.5501836644801195, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6703114503366376
[junit4] 2> 1347913 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@39413843 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@68d4d412; maxCacheMB=0.025390625 maxMergeSizeMB=0.826171875)),segFN=segments_1,generation=1}
[junit4] 2> 1347913 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1347913 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@28601707[collection1] main
[junit4] 2> 1347914 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1347915 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1347915 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1347915 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1347915 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1347916 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1347916 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1347916 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1347916 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1347917 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1347917 T11051 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28601707[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1347918 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1347918 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1347918 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1347918 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1503092850535956480
[junit4] 2> 1347920 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1503092850535956480
[junit4] 2> 1347920 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1347921 T11050 n:127.0.0.1:36903_u_mbs%2Fzf x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1347922 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36903/u_mbs/zf collection:collection1 shard:shard2
[junit4] 2> 1347922 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2
[junit4] 2> 1347922 T10984 n:127.0.0.1:36903_u_mbs%2Fzf oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1347923 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1347928 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1347929 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1347929 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1347930 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3000 name=collection1 org.apache.solr.core.SolrCore@2952508d url=http://127.0.0.1:36903/u_mbs/zf/collection1 node=127.0.0.1:36903_u_mbs%2Fzf C3000_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:36903/u_mbs/zf, node_name=127.0.0.1:36903_u_mbs%2Fzf, core=collection1, state=down}
[junit4] 2> 1347930 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 C3000 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36903/u_mbs/zf/collection1/
[junit4] 2> 1347930 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1"} current state version: 5
[junit4] 2> 1347931 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
[junit4] 2> 1347931 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
[junit4] 2> 1347932 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2
[junit4] 2> 1347930 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 C3000 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1347933 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 C3000 oasc.SyncStrategy.syncToMe http://127.0.0.1:36903/u_mbs/zf/collection1/ has no replicas
[junit4] 2> 1347934 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36903/u_mbs/zf/collection1/ shard2
[junit4] 2> 1347934 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1347937 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1347938 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:36903/u_mbs/zf",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 5
[junit4] 2> 1347938 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
[junit4] 2> 1347939 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
[junit4] 2> 1347939 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3
[junit4] 2> 1347941 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:36903/u_mbs/zf/collection1/ and leader is http://127.0.0.1:36903/u_mbs/zf/collection1/
[junit4] 2> 1347941 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36903/u_mbs/zf
[junit4] 2> 1347941 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1347941 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1347941 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1347942 T11018 n:127.0.0.1:43567_u_mbs%2Fzf oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1347943 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "base_url":"http://127.0.0.1:36903/u_mbs/zf",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:36903_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1"} current state version: 5
[junit4] 2> 1347944 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "base_url":"http://127.0.0.1:36903/u_mbs/zf",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "node_name":"127.0.0.1:36903_u_mbs%2Fzf",
[junit4] 2> "roles":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1"}
[junit4] 2> 1347945 T11019 n:127.0.0.1:43567_u_mbs%2Fzf oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
[junit4] 2> 1347945 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
[junit4] 2> 1347946 T11054 n:127.0.0.1:36903_u_mbs%2Fzf c:collection1 s:shard2 x:collection1 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 1347947 T11049 n:127.0.0.1:36903_u_mbs%2Fzf oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 1348217 T10984 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/cores/collection1
[junit4] 2> 1348218 T10984 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001
[junit4] 2> 1348219 T10984 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1348222 T10984 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@206cf1fc{/u_mbs/zf,null,AVAILABLE}
[junit4] 2> 1348222 T10984 oejs.AbstractConnector.doStart Started ServerConnector@424ec98e{HTTP/1.1}{127.0.0.1:38125}
[junit4] 2> 1348222 T10984 oejs.Server.doStart Started @1349657ms
[junit4] 2> 1348223 T10984 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/cores, solrconfig=solrconfig.xml, hostContext=/u_mbs/zf, hostPort=38125}
[junit4] 2> 1348223 T10984 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@7b3cb2c6
[junit4] 2> 1348223 T10984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/'
[junit4] 2> 1348238 T10984 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/solr.xml
[junit4] 2> 1348251 T10984 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/cores
[junit4] 2> 1348252 T10984 oasc.CoreContainer.<init> New CoreContainer 1768574986
[junit4] 2> 1348252 T10984 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/]
[junit4] 2> 1348252 T10984 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/lib
[junit4] 2> 1348252 T10984 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest 8EE5315ABA299BC9-001/shard-2-001/lib).
[junit4] 2> 1348260 T10984 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1348262 T10984 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1348262 T10984 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[j
[...truncated too long message...]
e.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> NOTE: All tests run in this JVM: [SearchHandlerTest, TestMacros, StatsComponentTest, AutoCommitTest, SSLMigrationTest, TestSortingResponseWriter, TestSystemIdResolver, PrimUtilsTest, BlockDirectoryTest, TestTrackingShardHandlerFactory, TestBlendedInfixSuggestions, DistributedFacetPivotSmallAdvancedTest, OverseerCollectionProcessorTest, ResponseLogComponentTest, DeleteShardTest, QueryEqualityTest, PolyFieldTest, HttpPartitionTest, TestSolrQueryParser, TestFastOutputStream, RecoveryZkTest, HdfsBasicDistributedZkTest, FileBasedSpellCheckerTest, TestGroupingSearch, BadIndexSchemaTest, DistributedVersionInfoTest, TestFieldCollectionResource, PathHierarchyTokenizerFactoryTest, TestSort, SoftAutoCommitTest, TestMissingGroups, TestQueryUtils, TestPerFieldSimilarity, TestClusterStateMutator, CoreAdminRequestStatusTest, SaslZkACLProviderTest, TestSolrConfigHandlerCloud, TestQueryTypes, HdfsCollectionsAPIDistributedZkTest, XsltUpdateRequestHandlerTest, CloudMLTQParserTest, TestRestoreCore, TestSolrConfigHandlerConcurrent, TestSolrConfigHandler, RegexBoostProcessorTest, TestRecovery, TestDistributedStatsComponentCardinality, TermVectorComponentDistributedTest, TestQuerySenderNoQuery, TestLuceneMatchVersion, OpenExchangeRatesOrgProviderTest, TestConfigOverlay, TestAddFieldRealTimeGet, LoggingHandlerTest, TestZkChroot, TestAuthenticationFramework, DirectSolrSpellCheckerTest, TestUtils, TestCollectionAPI, XmlUpdateRequestHandlerTest, SolrCoreCheckLockOnStartupTest, SegmentsInfoRequestHandlerTest, TestCloudPivotFacet, AnalysisErrorHandlingTest, NumericFieldsTest, MoreLikeThisHandlerTest, FullSolrCloudDistribCmdsTest, HdfsBasicDistributedZk2Test, TestPseudoReturnFields, TestBulkSchemaConcurrent, TestBinaryResponseWriter, ZkStateWriterTest, TestRecoveryHdfs, SuggesterTest, TestFieldTypeResource, ChangedSchemaMergeTest, TestManagedSchemaFieldResource, HdfsNNFailoverTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest]
[junit4] 2> 8329895 T11109 n:127.0.0.1:39787_u_mbs%2Fzf oasc.SolrException.log ERROR null:org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work/qnr-0000000024
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:412)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:224)
[junit4] 2> at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:168)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] ERROR 0.00s J2 | CollectionsAPIDistributedZkTest (suite) <<<
[junit4] > Throwable #1: java.lang.OutOfMemoryError: Java heap space
[junit4] Completed [497/497] on J2 in 42864.90s, 1 test, 1 error <<< FAILURES!
[...truncated 11 lines...]
[junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J2-20150604_191457_281.sysout
[junit4] >>> JVM J2: stdout (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps/java_pid28391.hprof ...
[junit4] Heap dump file created [660913182 bytes in 5.704 secs]
[junit4] <<< JVM J2: EOF ----
[...truncated 10 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 497 suites, 1974 tests, 1 suite-level error, 52 ignored (10 assumptions)
Total time: 860 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 1295455181 bytes
Compression is 0.0%
Took 3 min 47 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 10 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/10/
4 tests failed.
REGRESSION: org.apache.lucene.index.Test4GBStoredFields.test
Error Message:
No space left on device
Stack Trace:
java.io.IOException: No space left on device
at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:330840B92FD97112]:0)
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:142)
at org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:157)
at org.apache.lucene.codecs.compressing.LZ4.encodeLastLiterals(LZ4.java:162)
at org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:252)
at org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:235)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:165)
at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsWriter.finishDocument(AssertingStoredFieldsFormat.java:137)
at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:270)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:311)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142)
at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose
Error Message:
No space left on device
Stack Trace:
java.io.IOException: No space left on device
at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:9CADDEB25233CC1E]:0)
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80)
at org.apache.lucene.store.MockIndexOutputWrapper.getChecksum(MockIndexOutputWrapper.java:172)
at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:310)
at org.apache.lucene.codecs.lucene50.Lucene50NormsConsumer.close(Lucene50NormsConsumer.java:337)
at org.apache.lucene.codecs.asserting.AssertingNormsFormat$AssertingNormsConsumer.close(AssertingNormsFormat.java:78)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:96)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:83)
at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:211)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:92)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:420)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:512)
at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:382)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:475)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142)
at org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose(TestIndexWriterMerging.java:394)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.lucene.index.TestNRTThreads.testNRTThreads
Error Message:
No space left on device
Stack Trace:
java.io.IOException: No space left on device
at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:20856B78C0DE0A81]:0)
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:142)
at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:86)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:988)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:758)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:1066)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:1027)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:448)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3953)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3534)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1817)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86)
at org.apache.lucene.index.TestNRTThreads.doSearching(TestNRTThreads.java:76)
at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase.runTest(ThreadedIndexingAndSearchingTestCase.java:530)
at org.apache.lucene.index.TestNRTThreads.testNRTThreads(TestNRTThreads.java:149)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: TEST-org.apache.lucene.search.TestNumericRangeQuery64.xml.<init>
Error Message:
Stack Trace:
Test report file /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/TEST-org.apache.lucene.search.TestNumericRangeQuery64.xml was length 0
Build Log:
[...truncated 692 lines...]
[junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=Test4GBStoredFields -Dtests.method=test -Dtests.seed=BB5C7F6381251CEA -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr_BA_#Latn -Dtests.timezone=CST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 65.3s J1 | Test4GBStoredFields.test <<<
[junit4] > Throwable #1: java.io.IOException: No space left on device
[junit4] > at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:330840B92FD97112]:0)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
[junit4] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
[junit4] > at sun.nio.ch.IOUtil.write(IOUtil.java:65)
[junit4] > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
[junit4] > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
[junit4] > at java.nio.channels.Channels.writeFully(Channels.java:101)
[junit4] > at java.nio.channels.Channels.access$000(Channels.java:61)
[junit4] > at java.nio.channels.Channels$1.write(Channels.java:174)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
[junit4] > at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
[junit4] > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
[junit4] > at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:142)
[junit4] > at org.apache.lucene.codecs.compressing.LZ4.encodeLiterals(LZ4.java:157)
[junit4] > at org.apache.lucene.codecs.compressing.LZ4.encodeLastLiterals(LZ4.java:162)
[junit4] > at org.apache.lucene.codecs.compressing.LZ4.compress(LZ4.java:252)
[junit4] > at org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor.compress(CompressionMode.java:164)
[junit4] > at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:235)
[junit4] > at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:165)
[junit4] > at org.apache.lucene.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsWriter.finishDocument(AssertingStoredFieldsFormat.java:137)
[junit4] > at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:270)
[junit4] > at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:311)
[junit4] > at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
[junit4] > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:458)
[junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
[junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142)
[junit4] > at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/J1/temp/lucene.index.Test4GBStoredFields BB5C7F6381251CEA-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=sr_BA_#Latn, timezone=CST
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=277090024,total=311951360
[junit4] 2> NOTE: All tests run in this JVM: [TestMatchAllDocsQuery, TestPayloadExplanations, TestMixedCodecs, TestFilterIterator, TestLucene50CompoundFormat, TestApproximationSearchEquivalence, TestNumericTokenStream, TestSegmentReader, TestDirectory, TestIndexWriterOutOfMemory, TestDocValues, TestLock, TestIndexWriterConfig, TestLongBitSet, TestTermVectorsWriter, TestNoDeletionPolicy, TestIsCurrent, TestTermRangeFilter, TestTimSorterWorstCase, TestTragicIndexWriterDeadlock, TestTrackingDirectoryWrapper, Test4GBStoredFields]
[junit4] Completed [121/399] on J1 in 65.38s, 1 test, 1 error <<< FAILURES!
[...truncated 4 lines...]
[junit4] Suite: org.apache.lucene.index.TestNRTThreads
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestNRTThreads -Dtests.method=testNRTThreads -Dtests.seed=BB5C7F6381251CEA -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=de_CH -Dtests.timezone=America/Winnipeg -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 300s J2 | TestNRTThreads.testNRTThreads <<<
[junit4] > Throwable #1: java.io.IOException: No space left on device
[junit4] > at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:20856B78C0DE0A81]:0)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
[junit4] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
[junit4] > at sun.nio.ch.IOUtil.write(IOUtil.java:65)
[junit4] > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
[junit4] > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
[junit4] > at java.nio.channels.Channels.writeFully(Channels.java:101)
[junit4] > at java.nio.channels.Channels.access$000(Channels.java:61)
[junit4] > at java.nio.channels.Channels$1.write(Channels.java:174)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
[junit4] > at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
[junit4] > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4] > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
[junit4] > at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:53)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:142)
[junit4] > at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:73)
[junit4] > at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
[junit4] > at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:86)
[junit4] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:988)
[junit4] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:758)
[junit4] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:1066)
[junit4] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:1027)
[junit4] > at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:448)
[junit4] > at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:198)
[junit4] > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
[junit4] > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:193)
[junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:95)
[junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3953)
[junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3534)
[junit4] > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
[junit4] > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1817)
[junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)
[junit4] > at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:86)
[junit4] > at org.apache.lucene.index.TestNRTThreads.doSearching(TestNRTThreads.java:76)
[junit4] > at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase.runTest(ThreadedIndexingAndSearchingTestCase.java:530)
[junit4] > at org.apache.lucene.index.TestNRTThreads.testNRTThreads(TestNRTThreads.java:149)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/J2/temp/lucene.index.TestNRTThreads BB5C7F6381251CEA-001
[junit4] 2> Jun 03, 2015 3:08:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=28073, maxDocsPerChunk=5, blockSize=5), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=28073, blockSize=5)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {extra33=DFR I(F)BZ(0.3), extra1=DFR I(ne)3(800.0), extra11=DFR I(ne)B3(800.0), extra2=IB SPL-D2, extra20=DFR I(F)L2, extra21=DFR I(n)L3(800.0), extra28=DFR I(n)B1, extra30=DFR I(n)1, body=IB SPL-LZ(0.3), extra34=IB SPL-D1, extra39=DFR I(ne)Z(0.3), extra31=IB LL-D3(800.0), extra19=DFR I(ne)B1, extra27=DefaultSimilarity, extra35=DFR GBZ(0.3), titleTokenized=DFR I(n)1, extra16=DFR GL2, extra32=DFR I(F)B1, extra38=DFR I(n)B3(800.0), extra13=DFR I(ne)Z(0.3), extra0=DFR I(n)LZ(0.3), extra12=DFR I(n)B3(800.0), extra7=DFR I(ne)LZ(0.3), extra6=DFR I(n)B1, extra24=IB SPL-D2, extra9=DFR I(ne)2, extra23=DFR I(ne)3(800.0), extra3=IB LL-L1, extra17=DFR I(n)2, extra10=DFR I(F)3(800.0), extra36=DFR G2, extra29=DFR I(ne)LZ(0.3), extra26=IB SPL-L1, extra8=IB SPL-D3(800.0), extra15=DFR GL3(800.0), extra37=DFR I(ne)B3(800.0), extra18=IB LL-LZ(0.3), extra5=DefaultSimilarity, extra14=DFR I(F)L1, extra4=IB SPL-L1, extra22=DFR I(n)LZ(0.3), extra25=IB LL-L1}, locale=de_CH, timezone=America/Winnipeg
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=386071520,total=450363392
[junit4] 2> NOTE: All tests run in this JVM: [TestMergedIterator, TestBlockPostingsFormat, TestCheckIndex, TestSloppyMath, TestLucene50StoredFieldsFormatHighCompression, TestFieldCacheRangeFilter, TestIndexFileDeleter, TestDocsAndPositions, TestIOUtils, TestIndexableField, TestFieldCacheRewriteMethod, TestCachingWrapperQuery, Test2BPagedBytes, TestReusableStringReader, TestBufferedChecksum, TestSimilarityProvider, Test2BBinaryDocValues, TestElevationComparator, TestPayloads, TestSumDocFreq, Test2BPostingsBytes, TestMultiThreadTermVectors, TestFieldValueFilter, TestDateSort, TestInPlaceMergeSorter, TestOmitNorms, TestFilterLeafReader, TestConcurrentMergeScheduler, TestSimpleFSDirectory, TestDemo, TestTryDelete, TestLucene50NormsFormat, TestNRTThreads]
[junit4] Completed [123/399] on J2 in 300.61s, 1 test, 1 error <<< FAILURES!
[...truncated 1 lines...]
[junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexWriterMerging -Dtests.method=testNoWaitClose -Dtests.seed=BB5C7F6381251CEA -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=is_IS -Dtests.timezone=America/Indiana/Winamac -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.11s J0 | TestIndexWriterMerging.testNoWaitClose <<<
[junit4] > Throwable #1: java.io.IOException: No space left on device
[junit4] > at __randomizedtesting.SeedInfo.seed([BB5C7F6381251CEA:9CADDEB25233CC1E]:0)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[junit4] > at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
[junit4] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
[junit4] > at sun.nio.ch.IOUtil.write(IOUtil.java:65)
[junit4] > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
[junit4] > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
[junit4] > at java.nio.channels.Channels.writeFully(Channels.java:101)
[junit4] > at java.nio.channels.Channels.access$000(Channels.java:61)
[junit4] > at java.nio.channels.Channels$1.write(Channels.java:174)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
[junit4] > at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:268)
[junit4] > at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
[junit4] > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4] > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
[junit4] > at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80)
[junit4] > at org.apache.lucene.store.MockIndexOutputWrapper.getChecksum(MockIndexOutputWrapper.java:172)
[junit4] > at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:310)
[junit4] > at org.apache.lucene.codecs.lucene50.Lucene50NormsConsumer.close(Lucene50NormsConsumer.java:337)
[junit4] > at org.apache.lucene.codecs.asserting.AssertingNormsFormat$AssertingNormsConsumer.close(AssertingNormsFormat.java:78)
[junit4] > at org.apache.lucene.util.IOUtils.close(IOUtils.java:96)
[junit4] > at org.apache.lucene.util.IOUtils.close(IOUtils.java:83)
[junit4] > at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:211)
[junit4] > at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:92)
[junit4] > at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:420)
[junit4] > at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:512)
[junit4] > at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:382)
[junit4] > at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:475)
[junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1363)
[junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1142)
[junit4] > at org.apache.lucene.index.TestIndexWriterMerging.testNoWaitClose(TestIndexWriterMerging.java:394)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/J0/temp/lucene.index.TestIndexWriterMerging BB5C7F6381251CEA-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {termVector=Lucene50(blocksize=128), id=PostingsFormat(name=Memory doPackFST= true), count=PostingsFormat(name=Memory doPackFST= true), tvtest=FST50}, docValues:{}, sim=DefaultSimilarity, locale=is_IS, timezone=America/Indiana/Winamac
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=284572664,total=359137280
[junit4] 2> NOTE: All tests run in this JVM: [TestCodecUtil, TestRecyclingByteBlockAllocator, TestForUtil, TestRAMDirectory, TestFileSwitchDirectory, TestComplexExplanationsOfNonMatches, TestAutomatonQuery, TestNumericUtils, TestPostingsOffsets, TestDirectoryReader, TestLevenshteinAutomata, TestPositionIncrement, TestSegmentMerger, TestDeletionPolicy, TestBinaryDocument, TestNeverDelete, TestTermdocPerf, TestReaderClosed, TestExceedMaxTermLength, TestBooleanMinShouldMatch, TestIndexWriterMaxDocs, TestDirectPacked, TestFilterSpans, TestSmallFloat, TestIndexWriter, TestSearcherManager, TestSnapshotDeletionPolicy, TestScorerPerf, TestMaxTermFrequency, TestRateLimiter, TestNeedsScores, Test2BNumericDocValues, TestNot, TestBooleanQueryVisitSubscorers, TestPrefixCodedTerms, TestSpanExplanations, TestFilteredSearch, TestByteBlockPool, TestIndexWriterLockRelease, TestIndexWriterWithThreads, TestIndexInput, TestBytesRefHash, TestFlex, TestCachingWrapperFilter, TestDoc, TestDisjunctionMaxQuery, TestFieldsReader, TestTermVectorsReader, TestSpanOrQuery, TestSimpleAttributeImpl, TestSegmentTermDocs, TestIndexWriterCommit, TestPerSegmentDeletes, TestNRTReaderCleanup, TestDocumentsWriterDeleteQueue, TestWildcardRandom, TestArrayUtil, TestDemoParallelLeafReader, TestNoMergePolicy, TestSimilarityBase, TestFieldMaskingSpanQuery, TestSearchForDuplicates, TestFixedBitDocIdSet, TestCrashCausesCorruptIndex, TestFieldValueQuery, TestShardSearching, TestPayloadSpans, TestIndexWriterMerging]
[junit4] Completed [124/399] on J0 in 4.42s, 6 tests, 1 error <<< FAILURES!
[...truncated 4 lines...]
[junit4] JVM J2: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J2-20150603_145501_080.syserr
[junit4] >>> JVM J2: stderr (verbatim) ----
[junit4] WARN: Unhandled exception in event serialization. -> com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.JsonIOException: java.io.IOException: No space left on device
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:514)
[junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
[junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
[junit4] at java.io.PrintStream.flush(PrintStream.java:338)
[junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
[junit4] at java.io.PrintStream.write(PrintStream.java:482)
[junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
[junit4] at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
[junit4] at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
[junit4] at java.io.PrintStream.write(PrintStream.java:527)
[junit4] at java.io.PrintStream.print(PrintStream.java:669)
[junit4] at java.io.PrintStream.println(PrintStream.java:823)
[junit4] at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
[junit4] at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:697)
[junit4] at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:703)
[junit4] at java.lang.Throwable.printStackTrace(Throwable.java:667)
[junit4] at java.lang.Throwable.printStackTrace(Throwable.java:643)
[junit4] at java.lang.Throwable.printStackTrace(Throwable.java:634)
[junit4] at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:300)
[junit4] Caused by: java.io.IOException: No space left on device
[junit4] at java.io.FileOutputStream.writeBytes(Native Method)
[junit4] at java.io.FileOutputStream.write(FileOutputStream.java:345)
[junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
[junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
[junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.string(JsonWriter.java:535)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:364)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:626)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:578)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.Streams.write(Streams.java:67)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:98)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:66)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:82)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4] ... 21 more
[junit4] <<< JVM J2: EOF ----
[...truncated 4 lines...]
[junit4] JVM J0: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J0-20150603_145501_080.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] WARN: Unhandled exception in event serialization. -> com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.JsonIOException: java.io.IOException: No space left on device
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:514)
[junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:131)
[junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:181)
[junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:763)
[junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:851)
[junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
[junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
[junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
[junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
[junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] at java.lang.Thread.run(Thread.java:745)
[junit4] Caused by: java.io.IOException: No space left on device
[junit4] at java.io.FileOutputStream.writeBytes(Native Method)
[junit4] at java.io.FileOutputStream.write(FileOutputStream.java:345)
[junit4] at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
[junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
[junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.string(JsonWriter.java:535)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:364)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:626)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:578)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.Streams.write(Streams.java:67)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:98)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:66)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:82)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:66)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:82)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:66)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:82)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4] ... 28 more
[junit4] <<< JVM J0: EOF ----
[...truncated 8 lines...]
[junit4] ERROR: JVM J0 ended with an exception, command line: /x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=BB5C7F6381251CEA -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=5.2.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=5.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/test:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J0-20150603_145501_080.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J0-20150603_145501_080.suites
[junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
[junit4] ERROR: JVM J1 ended with an exception, command line: /x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=BB5C7F6381251CEA -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=5.2.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=5.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/test:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J1-20150603_145501_080.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J1-20150603_145501_080.suites
[junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
[junit4] ERROR: JVM J2 ended with an exception, command line: /x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=BB5C7F6381251CEA -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=5.2.0 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/tests.policy -Dtests.LUCENE_VERSION=5.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/test:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J2-20150603_145501_080.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/test/temp/junit4-J2-20150603_145501_080.suites
[junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
Total time: 14 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 1204916 bytes
Compression is 0.0%
Took 0.66 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 9 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/9/
1 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test
Error Message:
KeeperErrorCode = Session expired for /live_nodes
Stack Trace:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /live_nodes
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:325)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:325)
at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:531)
at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:269)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkForMissingCollection(AbstractFullDistribZkTestBase.java:1820)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:257)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11147 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/init-core-data-001
[junit4] 2> 1327967 T5842 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 1327967 T5842 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /dew/z
[junit4] 2> 1327972 T5842 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1327972 T5843 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1327972 T5843 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1328072 T5842 oasc.ZkTestServer.run start zk server on port:38921
[junit4] 2> 1328073 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1328073 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328076 T5850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58a0ce20 name:ZooKeeperConnection Watcher:127.0.0.1:38921 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328076 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328076 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1328076 T5842 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1328078 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1328079 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328080 T5853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46dbbe9e name:ZooKeeperConnection Watcher:127.0.0.1:38921/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328080 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328080 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1328080 T5842 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1328082 T5842 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1328084 T5842 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1328085 T5842 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1328087 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1328087 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1328090 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1328090 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1328093 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1328093 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1328094 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1328095 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1328097 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1328097 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1328099 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1328099 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1328101 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1328102 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1328103 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1328104 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1328105 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1328106 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1328107 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1328108 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1328110 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1328110 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1328112 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1328113 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328114 T5856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19dada30 name:ZooKeeperConnection Watcher:127.0.0.1:38921 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328114 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328115 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1328115 T5842 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1328116 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1328117 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328118 T5859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ff2bbb1 name:ZooKeeperConnection Watcher:127.0.0.1:38921/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328119 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328119 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1328119 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
[junit4] 2> 1328120 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
[junit4] 2> 1328122 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
[junit4] 2> 1328122 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
[junit4] 2> 1328124 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
[junit4] 2> 1328124 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/enumsConfig.xml
[junit4] 2> 1328126 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1328126 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1328128 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
[junit4] 2> 1328128 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
[junit4] 2> 1328130 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
[junit4] 2> 1328131 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
[junit4] 2> 1328132 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
[junit4] 2> 1328132 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
[junit4] 2> 1328134 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
[junit4] 2> 1328134 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
[junit4] 2> 1328136 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 1328137 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 1328138 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
[junit4] 2> 1328139 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
[junit4] 2> 1328140 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
[junit4] 2> 1328140 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
[junit4] 2> 1328142 T5842 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
[junit4] 2> 1328142 T5842 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
[junit4] 2> 1328423 T5842 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1
[junit4] 2> 1328424 T5842 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1328426 T5842 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@739a5c42{/dew/z,null,AVAILABLE}
[junit4] 2> 1328430 T5842 oejs.AbstractConnector.doStart Started ServerConnector@1b9894f8{HTTP/1.1}{127.0.0.1:46077}
[junit4] 2> 1328430 T5842 oejs.Server.doStart Started @1329968ms
[junit4] 2> 1328430 T5842 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=46077, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores, hostContext=/dew/z}
[junit4] 2> 1328431 T5842 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4dfd245f
[junit4] 2> 1328431 T5842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/'
[junit4] 2> 1328446 T5842 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/solr.xml
[junit4] 2> 1328459 T5842 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores
[junit4] 2> 1328460 T5842 oasc.CoreContainer.<init> New CoreContainer 1681529859
[junit4] 2> 1328460 T5842 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/]
[junit4] 2> 1328460 T5842 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/lib
[junit4] 2> 1328460 T5842 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/lib).
[junit4] 2> 1328468 T5842 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1328469 T5842 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1328470 T5842 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1328470 T5842 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1328470 T5842 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1328470 T5842 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38921/solr
[junit4] 2> 1328470 T5842 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1328471 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1328471 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328473 T5873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62188b3a name:ZooKeeperConnection Watcher:127.0.0.1:38921 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328473 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328473 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1328475 T5842 n:127.0.0.1:46077_dew%2Fz oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328476 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48cbdf10 name:ZooKeeperConnection Watcher:127.0.0.1:38921/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328477 T5842 n:127.0.0.1:46077_dew%2Fz oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328478 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1328481 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1328486 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1328490 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1328494 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1328498 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1328499 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1328505 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1328508 T5842 n:127.0.0.1:46077_dew%2Fz oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46077_dew%2Fz
[junit4] 2> 1328510 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46077_dew%2Fz
[junit4] 2> 1328513 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1328516 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1328518 T5842 n:127.0.0.1:46077_dew%2Fz oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1328520 T5842 n:127.0.0.1:46077_dew%2Fz oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46077_dew%2Fz
[junit4] 2> 1328520 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1328521 T5842 n:127.0.0.1:46077_dew%2Fz oasc.Overseer.start Overseer (id=93932907826511877-127.0.0.1:46077_dew%2Fz-n_0000000000) starting
[junit4] 2> 1328524 T5842 n:127.0.0.1:46077_dew%2Fz oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1328534 T5842 n:127.0.0.1:46077_dew%2Fz oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1328536 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1328537 T5878 n:127.0.0.1:46077_dew%2Fz oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1328546 T5842 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1328549 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1328550 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1328550 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores
[junit4] 2> 1328552 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {schema=schema.xml, loadOnStartup=true, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1, transient=false, config=solrconfig.xml, coreNodeName=, shard=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/, collection=control_collection, name=collection1, dataDir=data/}
[junit4] 2> 1328552 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/
[junit4] 2> 1328552 T5842 n:127.0.0.1:46077_dew%2Fz oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1328554 T5880 n:127.0.0.1:46077_dew%2Fz c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1328554 T5880 n:127.0.0.1:46077_dew%2Fz c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1328555 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1328555 T5880 n:127.0.0.1:46077_dew%2Fz oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1328556 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1:46077/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":"127.0.0.1:46077_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null} current state version: 0
[junit4] 2> 1328556 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1:46077/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":"127.0.0.1:46077_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null}
[junit4] 2> 1328557 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 1328557 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1328559 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1329555 T5880 n:127.0.0.1:46077_dew%2Fz oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1329555 T5880 n:127.0.0.1:46077_dew%2Fz oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1329556 T5880 n:127.0.0.1:46077_dew%2Fz oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1329556 T5880 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1329557 T5880 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1329557 T5880 n:127.0.0.1:46077_dew%2Fz oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/'
[junit4] 2> 1329574 T5880 n:127.0.0.1:46077_dew%2Fz oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1329581 T5880 n:127.0.0.1:46077_dew%2Fz oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1329598 T5880 n:127.0.0.1:46077_dew%2Fz oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 1329642 T5880 n:127.0.0.1:46077_dew%2Fz oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1329643 T5880 n:127.0.0.1:46077_dew%2Fz oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1329649 T5880 n:127.0.0.1:46077_dew%2Fz oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1329874 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1329882 T5880 n:127.0.0.1:46077_dew%2Fz oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1329884 T5880 n:127.0.0.1:46077_dew%2Fz oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1329894 T5880 n:127.0.0.1:46077_dew%2Fz oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1329898 T5880 n:127.0.0.1:46077_dew%2Fz oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1329901 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1329901 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1329902 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1329902 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1329903 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1329903 T5880 n:127.0.0.1:46077_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1329903 T5880 n:127.0.0.1:46077_dew%2Fz oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1329903 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1329904 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1329904 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@733867d
[junit4] 2> 1329905 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/data
[junit4] 2> 1329905 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/data/index/
[junit4] 2> 1329905 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1329906 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/control-001/cores/collection1/data/index
[junit4] 2> 1329906 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=51.87109375, floorSegmentMB=1.416015625, forceMergeDeletesPctAllowed=16.86291416978424, segmentsPerTier=36.0, maxCFSSegmentSizeMB=0.33203125, noCFSRatio=0.10451930635357823
[junit4] 2> 1329907 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@61056d26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2b410b8d),segFN=segments_1,generation=1}
[junit4] 2> 1329907 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1329913 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1329914 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1329914 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1329914 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1329915 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1329915 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1329916 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1329916 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1329916 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1329920 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1329922 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1329923 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1329925 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1329931 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/file,/admin/system,/get,/config,/admin/properties,/admin/segments,/update,/admin/plugins,/update/json,/update/csv,/admin/threads,/admin/logging,/admin/ping,/replication,/schema,standard,/admin/luke,/admin/mbeans,/update/json/docs
[junit4] 2> 1329932 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1329933 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1329933 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1329934 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1329934 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1329935 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=97.8740234375, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=3.892508738450504, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7843365551529103
[junit4] 2> 1329935 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@61056d26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2b410b8d),segFN=segments_1,generation=1}
[junit4] 2> 1329936 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1329936 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@10bd24ed[collection1] main
[junit4] 2> 1329936 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1329937 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1329937 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1329938 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1329938 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1329938 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1329939 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1329939 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1329939 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1329939 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1329940 T5881 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10bd24ed[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1329940 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1329940 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1329941 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1329941 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502926527286738944
[junit4] 2> 1329943 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1502926527286738944
[junit4] 2> 1329943 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1329944 T5880 n:127.0.0.1:46077_dew%2Fz x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1329945 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46077/dew/z collection:control_collection shard:shard1
[junit4] 2> 1329945 T5842 n:127.0.0.1:46077_dew%2Fz oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 1329945 T5842 n:127.0.0.1:46077_dew%2Fz oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1329947 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1329947 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1329948 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1329949 T5887 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@522c207e name:ZooKeeperConnection Watcher:127.0.0.1:38921/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1329950 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1329950 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1329950 T5842 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1329953 T5842 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1329954 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1329955 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1329956 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1329956 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C6833 name=collection1 org.apache.solr.core.SolrCore@74277be7 url=http://127.0.0.1:46077/dew/z/collection1 node=127.0.0.1:46077_dew%2Fz C6833_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:46077/dew/z, state=down, node_name=127.0.0.1:46077_dew%2Fz, core=collection1}
[junit4] 2> 1329957 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 C6833 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46077/dew/z/collection1/
[junit4] 2> 1329957 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 C6833 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1329957 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> 1329957 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 C6833 oasc.SyncStrategy.syncToMe http://127.0.0.1:46077/dew/z/collection1/ has no replicas
[junit4] 2> 1329957 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46077/dew/z/collection1/ shard1
[junit4] 2> 1329958 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1329965 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1329966 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:46077/dew/z",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 1330070 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1330070 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1330118 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:46077/dew/z/collection1/ and leader is http://127.0.0.1:46077/dew/z/collection1/
[junit4] 2> 1330119 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46077/dew/z
[junit4] 2> 1330119 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1330119 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1330119 T5884 n:127.0.0.1:46077_dew%2Fz c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1330120 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1330121 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:46077/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1:46077_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1"} current state version: 2
[junit4] 2> 1330122 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:46077/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1:46077_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1"}
[junit4] 2> 1330225 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1330225 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1330226 T5842 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1
[junit4] 2> 1330227 T5842 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001
[junit4] 2> 1330227 T5842 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1330230 T5842 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@6890837e{/dew/z,null,AVAILABLE}
[junit4] 2> 1330230 T5842 oejs.AbstractConnector.doStart Started ServerConnector@6eb67d0{HTTP/1.1}{127.0.0.1:53421}
[junit4] 2> 1330230 T5842 oejs.Server.doStart Started @1331768ms
[junit4] 2> 1330231 T5842 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=53421, hostContext=/dew/z, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores, solrconfig=solrconfig.xml}
[junit4] 2> 1330231 T5842 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4dfd245f
[junit4] 2> 1330231 T5842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/'
[junit4] 2> 1330247 T5842 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/solr.xml
[junit4] 2> 1330261 T5842 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores
[junit4] 2> 1330261 T5842 oasc.CoreContainer.<init> New CoreContainer 378654163
[junit4] 2> 1330261 T5842 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/]
[junit4] 2> 1330261 T5842 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/lib
[junit4] 2> 1330262 T5842 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/lib).
[junit4] 2> 1330269 T5842 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 1330271 T5842 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 1330271 T5842 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1330272 T5842 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1330272 T5842 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 1330272 T5842 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38921/solr
[junit4] 2> 1330272 T5842 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1330273 T5842 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1330274 T5842 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1330276 T5901 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47c0fc11 name:ZooKeeperConnection Watcher:127.0.0.1:38921 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1330276 T5842 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1330277 T5842 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1330279 T5842 n:127.0.0.1:53421_dew%2Fz oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1330280 T5904 n:127.0.0.1:53421_dew%2Fz oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@271008a7 name:ZooKeeperConnection Watcher:127.0.0.1:38921/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1330281 T5842 n:127.0.0.1:53421_dew%2Fz oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1330285 T5842 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1331289 T5842 n:127.0.0.1:53421_dew%2Fz oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53421_dew%2Fz
[junit4] 2> 1331290 T5842 n:127.0.0.1:53421_dew%2Fz oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53421_dew%2Fz
[junit4] 2> 1331294 T5842 n:127.0.0.1:53421_dew%2Fz oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1331296 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1331296 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1331297 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores
[junit4] 2> 1331298 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1, transient=false, collection=collection1, loadOnStartup=true, dataDir=data/, coreNodeName=, shard=, name=collection1, schema=schema.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/, config=solrconfig.xml}
[junit4] 2> 1331299 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/
[junit4] 2> 1331299 T5842 n:127.0.0.1:53421_dew%2Fz oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1331304 T5905 n:127.0.0.1:53421_dew%2Fz c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1331307 T5905 n:127.0.0.1:53421_dew%2Fz c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1331308 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1331309 T5905 n:127.0.0.1:53421_dew%2Fz oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1331310 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:53421/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":"127.0.0.1:53421_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null} current state version: 3
[junit4] 2> 1331310 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:53421/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "state":"down",
[junit4] 2> "node_name":"127.0.0.1:53421_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":null}
[junit4] 2> 1331310 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 1331311 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1331413 T5904 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1331413 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1331415 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1332309 T5905 n:127.0.0.1:53421_dew%2Fz oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1332309 T5905 n:127.0.0.1:53421_dew%2Fz oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1332310 T5905 n:127.0.0.1:53421_dew%2Fz oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1332310 T5905 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1332311 T5905 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1332311 T5905 n:127.0.0.1:53421_dew%2Fz oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/'
[junit4] 2> 1332325 T5905 n:127.0.0.1:53421_dew%2Fz oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1332331 T5905 n:127.0.0.1:53421_dew%2Fz oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1332343 T5905 n:127.0.0.1:53421_dew%2Fz oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 1332447 T5905 n:127.0.0.1:53421_dew%2Fz oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1332448 T5905 n:127.0.0.1:53421_dew%2Fz oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1332458 T5905 n:127.0.0.1:53421_dew%2Fz oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1333094 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1333104 T5905 n:127.0.0.1:53421_dew%2Fz oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1333106 T5905 n:127.0.0.1:53421_dew%2Fz oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1333115 T5905 n:127.0.0.1:53421_dew%2Fz oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1333144 T5905 n:127.0.0.1:53421_dew%2Fz oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1333160 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1333162 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1333162 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1333162 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1333168 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1333168 T5905 n:127.0.0.1:53421_dew%2Fz oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1333168 T5905 n:127.0.0.1:53421_dew%2Fz oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1333168 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1333168 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 1333169 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@733867d
[junit4] 2> 1333169 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/data
[junit4] 2> 1333170 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 1333170 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1333170 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 1333170 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=51.87109375, floorSegmentMB=1.416015625, forceMergeDeletesPctAllowed=16.86291416978424, segmentsPerTier=36.0, maxCFSSegmentSizeMB=0.33203125, noCFSRatio=0.10451930635357823
[junit4] 2> 1333172 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7b2275ce lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1364668a),segFN=segments_1,generation=1}
[junit4] 2> 1333172 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1333177 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1333178 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1333178 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1333178 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1333178 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1333179 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1333179 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1333179 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1333179 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1333182 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1333187 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1333189 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1333190 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1333199 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/file,/admin/system,/get,/config,/admin/properties,/admin/segments,/update,/admin/plugins,/update/json,/update/csv,/admin/threads,/admin/logging,/admin/ping,/replication,/schema,standard,/admin/luke,/admin/mbeans,/update/json/docs
[junit4] 2> 1333201 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1333202 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1333202 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1333203 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1333203 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1333219 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=97.8740234375, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=3.892508738450504, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7843365551529103
[junit4] 2> 1333219 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7b2275ce lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1364668a),segFN=segments_1,generation=1}
[junit4] 2> 1333220 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1333220 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@745675c[collection1] main
[junit4] 2> 1333220 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1333221 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1333221 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1333223 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1333223 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1333223 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1333224 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1333224 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1333224 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1333224 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1333225 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Looking up max value of version field to seed version buckets
[junit4] 2> 1333225 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1333225 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1333225 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502926530730262528
[junit4] 2> 1333228 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1502926530730262528
[junit4] 2> 1333228 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 1333228 T5905 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1333233 T5906 n:127.0.0.1:53421_dew%2Fz x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@745675c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1333234 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53421/dew/z collection:collection1 shard:shard1
[junit4] 2> 1333235 T5842 n:127.0.0.1:53421_dew%2Fz oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 1333235 T5842 n:127.0.0.1:53421_dew%2Fz oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1333236 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1333240 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1333241 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1333242 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> 1333243 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333243 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333244 T5904 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333252 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1333253 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C6834 name=collection1 org.apache.solr.core.SolrCore@6d005e64 url=http://127.0.0.1:53421/dew/z/collection1 node=127.0.0.1:53421_dew%2Fz C6834_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:53421/dew/z, state=down, node_name=127.0.0.1:53421_dew%2Fz, core=collection1}
[junit4] 2> 1333253 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 C6834 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53421/dew/z/collection1/
[junit4] 2> 1333253 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 C6834 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1333253 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 C6834 oasc.SyncStrategy.syncToMe http://127.0.0.1:53421/dew/z/collection1/ has no replicas
[junit4] 2> 1333253 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53421/dew/z/collection1/ shard1
[junit4] 2> 1333254 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1333262 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1333263 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:53421/dew/z",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 5
[junit4] 2> 1333366 T5904 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333366 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333366 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333408 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:53421/dew/z/collection1/ and leader is http://127.0.0.1:53421/dew/z/collection1/
[junit4] 2> 1333408 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53421/dew/z
[junit4] 2> 1333408 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1333408 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1333408 T5909 n:127.0.0.1:53421_dew%2Fz c:collection1 s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1333410 T5876 n:127.0.0.1:46077_dew%2Fz oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1333410 T5877 n:127.0.0.1:46077_dew%2Fz oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:53421/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1:53421_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1"} current state version: 6
[junit4] 2> 1333411 T5877 n:127.0.0.1:46077_dew%2Fz oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "base_url":"http://127.0.0.1:53421/dew/z",
[junit4] 2> "operation":"state",
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1:53421_dew%2Fz",
[junit4] 2> "core":"collection1",
[junit4] 2> "shard":"shard1"}
[junit4] 2> 1333513 T5904 n:127.0.0.1:53421_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333514 T5887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333514 T5876 n:127.0.0.1:46077_dew%2Fz oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1333861 T5842 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-2-001/cores/collection1
[junit4] 2> 1333862 T5842 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-2-001
[junit4] 2> 1333863 T5842 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 1333912 T5842 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5fd87962{/dew/z,null,AVAILABLE}
[junit4] 2> 1333913 T5842 oejs.AbstractConnector.doStart Started ServerConnector@7d094216{HTTP/1.1}{127.0.0.1:57066}
[junit4] 2> 1333913 T5842 oejs.Server.doStart Started @1335451ms
[junit4] 2> 1333914 T5842 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=57066, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-2-001/cores, hostContext=/dew/z, solrconfig=solrconfig.xml}
[junit4] 2> 1333914 T5842 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4dfd245f
[junit4] 2> 1333914 T5842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001/shard-2-001/'
[junit4] 2> 1333932 T5842 oasc.SolrXmlConfig.fromFile Loading container conf
[...truncated too long message...]
ient.executeMethod(HttpSolrClient.java:570)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 more
[junit4] > Caused by: java.net.SocketTimeoutException: Read timed out
[junit4] > at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:152)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:122)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
[junit4] > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
[junit4] > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
[junit4] > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
[junit4] > at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
[junit4] > ... 11 moreThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6321, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:46077/dew/z, http://127.0.0.1:53421/dew/z, http://127.0.0.1:38923/dew/z, http://127.0.0.1:57066/dew/z, http://127.0.0.1:34988/dew/z]
[junit4] > at __randomizedtesting.SeedInfo.seed([130D33018272A4CD]:0)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:46077/dew/z, http://127.0.0.1:53421/dew/z, http://127.0.0.1:38923/dew/z, http://127.0.0.1:57066/dew/z, http://127.0.0.1:34988/dew/z]
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
[junit4] > at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
[junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
[junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53421/dew/z: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
[junit4] > ... 7 more
[junit4] 2> 2434752 T5842 c:collection1 s:shard2 x:collection1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 130D33018272A4CD-001
[junit4] 2> 1106790 T5841 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene50, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_SA, timezone=America/Indiana/Indianapolis
[junit4] 2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.7.0_72 (64-bit)/cpus=4,threads=1,free=124552960,total=532152320
[junit4] 2> NOTE: All tests run in this JVM: [CurrencyFieldOpenExchangeTest, CachingDirectoryFactoryTest, TestJsonRequest, DocumentBuilderTest, ZkSolrClientTest, TestDynamicFieldResource, ShardRoutingTest, TestFiltering, HdfsLockFactoryTest, TestManagedStopFilterFactory, TestClassNameShortening, AnalysisAfterCoreReloadTest, TestRemoveLastDynamicCopyField, SpellCheckComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestManagedResourceStorage, TestJsonFacets, TestTolerantSearch, TestSolrXml, DistribJoinFromCollectionTest, UpdateParamsTest, CacheHeaderTest, ScriptEngineTest, SimpleMLTQParserTest, SuggesterTSTTest, TestComponentsName, HdfsWriteToMultipleCollectionsTest, TestLRUCache, TestCharFilters, SliceStateTest, OpenCloseCoreStressTest, ZkCLITest, VersionInfoTest, TestOverriddenPrefixQueryForCustomFieldType, TestCustomSort, TestFastWriter, DisMaxRequestHandlerTest, TestMiniSolrCloudCluster, TestCursorMarkWithoutUniqueKey, TestSearchPerf, RAMDirectoryFactoryTest, FieldAnalysisRequestHandlerTest, CopyFieldTest, TestShardHandlerFactory, TestCollationFieldDocValues, PreAnalyzedUpdateProcessorTest, VMParamsZkACLAndCredentialsProvidersTest, DistributedDebugComponentTest, TestCloudSchemaless, TestElisionMultitermQuery, TestAuthorizationFramework, TestNamedUpdateProcessors, SharedFSAutoReplicaFailoverTest, TestDefaultStatsCache, TestIndexSearcher, DistanceUnitsTest, IndexSchemaRuntimeFieldTest, ShowFileRequestHandlerTest, SolrPluginUtilsTest, TestSchemaResource, TestManagedSchemaFieldTypeResource, CoreAdminHandlerTest, DistributedMLTComponentTest, TestLFUCache, ClusterStateUpdateTest, TestCloudManagedSchema, ChaosMonkeySafeLeaderTest, TestDistributedGrouping, BadComponentTest, TestLRUStatsCache, JsonLoaderTest, TestChildDocTransformer, SpellingQueryConverterTest, HdfsUnloadDistributedZkTest, AliasIntegrationTest, AssignTest, EchoParamsTest, CollectionsAPIDistributedZkTest]
[junit4] Completed [379/497] on J1 in 1107.57s, 1 test, 1 error <<< FAILURES!
[...truncated 382 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 497 suites, 1974 tests, 1 error, 52 ignored (10 assumptions)
Total time: 156 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 993224922 bytes
Compression is 0.0%
Took 2 min 41 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 8 - Still Failing
Posted by Mark Miller <ma...@gmail.com>.
Uh oh...
On Tue, Jun 2, 2015 at 6:08 AM Apache Jenkins Server <
jenkins@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/8/
>
> 1 tests failed.
> REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
>
> Error Message:
> shard3 is not consistent. Got 1164 from
> http://127.0.0.1:35113/emwi/dt/collection1lastClient and got 1020 from
> http://127.0.0.1:42407/emwi/dt/collection1
>
> Stack Trace:
> java.lang.AssertionError: shard3 is not consistent. Got 1164 from
> http://127.0.0.1:35113/emwi/dt/collection1lastClient and got 1020 from
> http://127.0.0.1:42407/emwi/dt/collection1
> at
> __randomizedtesting.SeedInfo.seed([3E9C70166BCD829F:B6C84FCCC531EF67]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1293)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1272)
> at
> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:174)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
> at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
> at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 10742 lines...]
> [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
> [junit4] 2> Creating dataDir:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/init-core-data-001
> [junit4] 2> 2536136 T14073 oas.SolrTestCaseJ4.buildSSLConfig
> Randomized ssl (false) and clientAuth (true)
> [junit4] 2> 2536137 T14073
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /emwi/dt
> [junit4] 2> 2536142 T14073 oasc.ZkTestServer.run STARTING ZK TEST
> SERVER
> [junit4] 2> 2536146 T14074 oasc.ZkTestServer$2$1.setClientPort client
> port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 2536146 T14074
> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4] 2> 2536246 T14073 oasc.ZkTestServer.run start zk server on
> port:47741
> [junit4] 2> 2536246 T14073
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
> [junit4] 2> 2536262 T14073 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 2536273 T14081 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3c3c999a
> name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2536274 T14073 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 2536274 T14073 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
> [junit4] 2> 2536274 T14073 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 2536277 T14073
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
> [junit4] 2> 2536286 T14073 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 2536296 T14084 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2808fb2e
> name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2536296 T14073 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 2536297 T14073 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
> [junit4] 2> 2536297 T14073 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
> [junit4] 2> 2536299 T14073 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
> [junit4] 2> 2536376 T14073 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
> [junit4] 2> 2536377 T14073 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
> [junit4] 2> 2536378 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
> [junit4] 2> 2536379 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
> [junit4] 2> 2536381 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema15.xml
> to /configs/conf1/schema.xml
> [junit4] 2> 2536381 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
> [junit4] 2> 2536383 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 2536384 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 2536385 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
> [junit4] 2> 2536386 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
> [junit4] 2> 2536388 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
> [junit4] 2> 2536388 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
> [junit4] 2> 2536390 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
> [junit4] 2> 2536390 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
> [junit4] 2> 2536392 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
> [junit4] 2> 2536392 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
> [junit4] 2> 2536394 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
> [junit4] 2> 2536394 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
> [junit4] 2> 2536396 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 2536396 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 2536398 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
> [junit4] 2> 2536398 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
> [junit4] 2> 2536399 T14073 oasc.AbstractZkTestCase.putConfig put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
> [junit4] 2> 2536400 T14073 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
> [junit4] 2> 2536401 T14075 oazs.NIOServerCnxn.doIO WARN caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x14db3a1674a0001, likely client has closed socket
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
> [junit4] 2>
> [junit4] 2> 2536777 T14073 oas.SolrTestCaseJ4.writeCoreProperties
> Writing core.properties file to
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1
> [junit4] 2> 2536796 T14073 oejs.Server.doStart jetty-9.2.10.v20150310
> [junit4] 2> 2536982 T14073 oejsh.ContextHandler.doStart Started
> o.e.j.s.ServletContextHandler@a67b61d{/emwi/dt,null,AVAILABLE}
> [junit4] 2> 2536984 T14073 oejs.AbstractConnector.doStart Started
> ServerConnector@1ee52f41{HTTP/1.1}{127.0.0.1:43586}
> [junit4] 2> 2536984 T14073 oejs.Server.doStart Started @2538520ms
> [junit4] 2> 2536985 T14073 oascse.JettySolrRunner$1.lifeCycleStarted
> Jetty properties:
> {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores, hostPort=43586,
> hostContext=/emwi/dt,
> solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/tempDir-001/control/data}
> [junit4] 2> 2536985 T14073 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
> [junit4] 2> 2536985 T14073 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/'
> [junit4] 2> 2536999 T14073 oasc.SolrXmlConfig.fromFile Loading
> container configuration from
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/solr.xml
> [junit4] 2> 2537012 T14073 oasc.CorePropertiesLocator.<init>
> Config-defined core root directory:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores
> [junit4] 2> 2537013 T14073 oasc.CoreContainer.<init> New
> CoreContainer 2139689457
> [junit4] 2> 2537013 T14073 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/]
> [junit4] 2> 2537013 T14073 oasc.CoreContainer.load loading shared
> library:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/lib
> [junit4] 2> 2537013 T14073 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: lib (resolved
> as:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/lib).
> [junit4] 2> 2537020 T14073 oashc.HttpShardHandlerFactory.init created
> with socketTimeout : 90000,urlScheme : ,connTimeout :
> 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize :
> 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue :
> -1,fairnessPolicy : false,useRetries : false,
> [junit4] 2> 2537022 T14073 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=true
> [junit4] 2> 2537023 T14073 oasl.LogWatcher.createWatcher SLF4J impl
> is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 2537023 T14073 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 2537023 T14073 oasc.CoreContainer.load Node Name:
> 127.0.0.1
> [junit4] 2> 2537023 T14073 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:47741/solr
> [junit4] 2> 2537023 T14073 oasc.ZkController.checkChrootPath zkHost
> includes chroot
> [junit4] 2> 2537024 T14073
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
> [junit4] 2> 2537025 T14073 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 2537027 T14098 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@89ad624
> name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2537027 T14073 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 2537027 T14073 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
> [junit4] 2> 2537047 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ConnectionManager.waitForConnected Waiting for client to connect to
> ZooKeeper
> [junit4] 2> 2537056 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@51f61ec4
> name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2537056 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 2537057 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4] 2> 2537059 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4] 2> 2537061 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
> [junit4] 2> 2537063 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
> [junit4] 2> 2537064 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
> [junit4] 2> 2537066 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4] 2> 2537068 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4] 2> 2537069 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4] 2> 2537070 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.ZkController.createEphemeralLiveNode Register node as live in
> ZooKeeper:/live_nodes/127.0.0.1:43586_emwi%2Fdt
> [junit4] 2> 2537070 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43586
> _emwi%2Fdt
> [junit4] 2> 2537072 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer_elect
> [junit4] 2> 2537073 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4] 2> 2537074 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer.close Overseer (id=null) closing
> [junit4] 2> 2537075 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:43586_emwi%2Fdt
> [junit4] 2> 2537076 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4] 2> 2537077 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer.start Overseer
> (id=93928673048657923-127.0.0.1:43586_emwi%2Fdt-n_0000000000) starting
> [junit4] 2> 2537078 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4] 2> 2537083 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.OverseerAutoReplicaFailoverThread.<init> Starting
> OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
> [junit4] 2> 2537103 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4] 2> 2537103 T14103 n:127.0.0.1:43586_emwi%2Fdt
> oasc.OverseerCollectionProcessor.run Process current queue of collection
> creations
> [junit4] 2> 2537114 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 2537117 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin
> used.
> [junit4] 2> 2537117 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't
> exist. Skipping setup for authorization module.
> [junit4] 2> 2537118 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CorePropertiesLocator.discover Looking for core definitions underneath
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores
> [junit4] 2> 2537119 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {dataDir=data/,
> absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/, transient=false,
> name=collection1, shard=, coreNodeName=, loadOnStartup=true,
> collection=control_collection, config=solrconfig.xml, schema=schema.xml,
> instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1}
> [junit4] 2> 2537119 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CorePropertiesLocator.discoverUnder Found core collection1 in
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/
> [junit4] 2> 2537120 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CorePropertiesLocator.discover Found 1 core definitions
> [junit4] 2> 2537135 T14105 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection x:collection1 oasc.ZkController.publish publishing
> core=collection1 state=down collection=control_collection
> [junit4] 2> 2537135 T14105 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection x:collection1 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
> [junit4] 2> 2537136 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 2537136 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2537137 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "numShards":"1",
> [junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "shard":null,
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
> [junit4] 2> "state":"down"} current state version: 0
> [junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ReplicaMutator.updateState Update state numShards=1 message={
> [junit4] 2> "numShards":"1",
> [junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "shard":null,
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
> [junit4] 2> "state":"down"}
> [junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ClusterStateMutator.createCollection building a new cName:
> control_collection
> [junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 2537139 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538136 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.ZkController.waitForShardId waiting to find shard id in clusterstate
> for collection1
> [junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.ZkController.createCollectionZkNode Check for collection
> zkNode:control_collection
> [junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader.readConfigName Load collection config
> from:/collections/control_collection
> [junit4] 2> 2538138 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader.readConfigName path=/collections/control_collection
> configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 2538138 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/'
> [junit4] 2> 2538162 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Config.<init> loaded config solrconfig.xml with version 0
> [junit4] 2> 2538168 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
> [junit4] 2> 2538190 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
> [junit4] 2> 2538224 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 2538226 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.IndexSchema.readSchema Reading Solr Schema from
> /configs/conf1/schema.xml
> [junit4] 2> 2538231 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 2538562 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 2538564 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 2538565 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.FileExchangeRateProvider.reload Reloading exchange rates from file
> currency.xml
> [junit4] 2> 2538578 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oass.FileExchangeRateProvider.reload Reloading exchange rates from file
> currency.xml
> [junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt
> oasc.CoreContainer.create Creating SolrCore 'collection1' using
> configuration from collection control_collection
> [junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at
> [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/], dataDir=[null]
> [junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans
> to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41aacb08
> [junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.CachingDirectoryFactory.get return new directory for
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data
> [junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.getNewIndexDir New index directory detected: old=null
> new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data/index/
> [junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.initIndex WARN [collection1] Solr index directory
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data/index' doesn't
> exist. Creating new index...
> [junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.CachingDirectoryFactory.get return new directory for
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data/index
> [junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=18, maxMergeAtOnceExplicit=37,
> maxMergedSegmentMB=95.42578125, floorSegmentMB=0.2548828125,
> forceMergeDeletesPctAllowed=13.814119626493529, segmentsPerTier=18.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8506087588632382
> [junit4] 2> 2538613 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2>
> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
> [junit4] 2> 2538613 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 2538617 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "nodistrib"
> [junit4] 2> 2538617 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "dedupe"
> [junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "distrib-dup-test-chain-explicit"
> [junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 2538619 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 2538619 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain
> defined as default, creating implicit default
> [junit4] 2> 2538622 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2538623 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2538624 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2538625 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2538629 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.RequestHandlers.initHandlersFromConfig Registered paths:
> /admin/logging,/admin/threads,/admin/segments,/admin/ping,/admin/properties,/update/json,/update,standard,/config,/admin/file,/admin/system,/get,/replication,/admin/mbeans,/admin/plugins,/admin/luke,/update/json/docs,/update/csv,/schema
> [junit4] 2> 2538630 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.initStatsCache Using default statsCache cache:
> org.apache.solr.search.stats.LocalStatsCache
> [junit4] 2> 2538631 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.UpdateHandler.<init> Using UpdateLog implementation:
> org.apache.solr.update.UpdateLog
> [junit4] 2> 2538631 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
> numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=31, maxMergeAtOnceExplicit=10,
> maxMergedSegmentMB=1.3017578125, floorSegmentMB=0.5830078125,
> forceMergeDeletesPctAllowed=8.067275561099606, segmentsPerTier=19.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6133962934564373
> [junit4] 2> 2538633 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2>
> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
> [junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oass.SolrIndexSearcher.<init> Opening Searcher@60b77795[collection1] main
> [junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oascc.ZkStateReader.readConfigName Load collection config
> from:/collections/control_collection
> [junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oascc.ZkStateReader.readConfigName path=/collections/control_collection
> configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage
> for the RestManager with znodeBase: /configs/conf1
> [junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
> [junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.RestManager.init Initializing RestManager with initArgs: {}
> [junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.load Reading _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
> [junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json
> using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 2538637 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasr.RestManager.init Initializing 0 registered ManagedResources
> [junit4] 2> 2538637 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4] 2> 2538638 T14106 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.SolrCore.registerSearcher [collection1] Registered new searcher
> Searcher@60b77795[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> [junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max
> value of version field
> [junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of
> _version_ for 65536 version buckets from index
> [junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_,
> cannot seed version bucket highest value from index
> [junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in
> index or recent updates, using new clock 1502858771390529536
> [junit4] 2> 2538640 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version
> buckets with highest version 1502858771390529536
> [junit4] 2> 2538641 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
> [junit4] 2> 2538642 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1
> oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 2538643 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.register
> Register replica - core:collection1 address:http://127.0.0.1:43586/emwi/dt
> collection:control_collection shard:shard1
> [junit4] 2> 2538643 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oass.SolrDispatchFilter.init
> user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
> [junit4] 2> 2538644 T14073 n:127.0.0.1:43586_emwi%2Fdt
> oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 2538645 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath
> makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4] 2> 2538645 T14073
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
> [junit4] 2> 2538646 T14073 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 2538648 T14112 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1069b44e
> name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2538648 T14073 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 2538648 T14073 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
> [junit4] 2> 2538649 T14073
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 2538654 T14073 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
> [junit4] 2> 2538661 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
> [junit4] 2> 2538662 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2538663 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 2538664 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
> [junit4] 2> 2538664 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"control_collection"} current state
> version: 1
> [junit4] 2> ASYNC NEW_CORE C6968 name=collection1
> org.apache.solr.core.SolrCore@4b88ed77 url=
> http://127.0.0.1:43586/emwi/dt/collection1 node=127.0.0.1:43586_emwi%2Fdt
> C6968_STATE=coll:control_collection core:collection1 props:{base_url=
> http://127.0.0.1:43586/emwi/dt, core=collection1,
> node_name=127.0.0.1:43586_emwi%2Fdt, state=down}
> [junit4] 2> 2538664 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 C6968 oasc.SyncStrategy.sync
> Sync replicas to http://127.0.0.1:43586/emwi/dt/collection1/
> [junit4] 2> 2538665 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 C6968
> oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4] 2> 2538665 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 C6968
> oasc.SyncStrategy.syncToMe http://127.0.0.1:43586/emwi/dt/collection1/
> has no replicas
> [junit4] 2> 2538666 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:43586/emwi/dt/collection1/ shard1
> [junit4] 2> 2538666 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath
> makePath: /collections/control_collection/leaders/shard1
> [junit4] 2> 2538666 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538667 T14112 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538675 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2538677 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "state":"active"} current state version: 2
> [junit4] 2> 2538780 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538780 T14112 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538855 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.register We
> are http://127.0.0.1:43586/emwi/dt/collection1/ and leader is
> http://127.0.0.1:43586/emwi/dt/collection1/
> [junit4] 2> 2538856 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.register No
> LogReplay needed for core=collection1 baseURL=
> http://127.0.0.1:43586/emwi/dt
> [junit4] 2> 2538856 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery
> I am the leader, no recovery necessary
> [junit4] 2> 2538857 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.publish
> publishing core=collection1 state=active collection=control_collection
> [junit4] 2> 2538857 T14109 n:127.0.0.1:43586_emwi%2Fdt
> c:control_collection s:shard1 x:collection1 oasc.ZkController.publish
> numShards not found on descriptor - reading it from system property
> [junit4] 2> 2538862 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2538864 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "core_node_name":"core_node1",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
> [junit4] 2> "state":"active"} current state version: 3
> [junit4] 2> 2538865 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ReplicaMutator.updateState Update state numShards=3 message={
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "core_node_name":"core_node1",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
> [junit4] 2> "state":"active"}
> [junit4] 2> 2538969 T14112 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 2538969 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 1)
> [junit4] 2> 2539067 T14073 oas.SolrTestCaseJ4.writeCoreProperties
> Writing core.properties file to
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1
> [junit4] 2> 2539068 T14073
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001
> [junit4] 2> 2539069 T14073 oejs.Server.doStart jetty-9.2.10.v20150310
> [junit4] 2> 2539089 T14073 oejsh.ContextHandler.doStart Started
> o.e.j.s.ServletContextHandler@457c1274{/emwi/dt,null,AVAILABLE}
> [junit4] 2> 2539090 T14073 oejs.AbstractConnector.doStart Started
> ServerConnector@3c1774f0{HTTP/1.1}{127.0.0.1:60967}
> [junit4] 2> 2539090 T14073 oejs.Server.doStart Started @2540626ms
> [junit4] 2> 2539090 T14073 oascse.JettySolrRunner$1.lifeCycleStarted
> Jetty properties: {hostContext=/emwi/dt, hostPort=60967,
> coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores,
> solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/tempDir-001/jetty1, solrconfig=solrconfig.xml}
> [junit4] 2> 2539091 T14073 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
> [junit4] 2> 2539091 T14073 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/'
> [junit4] 2> 2539115 T14073 oasc.SolrXmlConfig.fromFile Loading
> container configuration from
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/solr.xml
> [junit4] 2> 2539148 T14073 oasc.CorePropertiesLocator.<init>
> Config-defined core root directory:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores
> [junit4] 2> 2539149 T14073 oasc.CoreContainer.<init> New
> CoreContainer 1270053320
> [junit4] 2> 2539149 T14073 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/]
> [junit4] 2> 2539149 T14073 oasc.CoreContainer.load loading shared
> library:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/lib
> [junit4] 2> 2539149 T14073 oasc.SolrResourceLoader.addToClassLoader
> WARN Can't find (or read) directory to add to classloader: lib (resolved
> as:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/lib).
> [junit4] 2> 2539165 T14073 oashc.HttpShardHandlerFactory.init created
> with socketTimeout : 90000,urlScheme : ,connTimeout :
> 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize :
> 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue :
> -1,fairnessPolicy : false,useRetries : false,
> [junit4] 2> 2539166 T14073 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=true
> [junit4] 2> 2539167 T14073 oasl.LogWatcher.createWatcher SLF4J impl
> is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 2539167 T14073 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 2539167 T14073 oasc.CoreContainer.load Node Name:
> 127.0.0.1
> [junit4] 2> 2539167 T14073 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:47741/solr
> [junit4] 2> 2539168 T14073 oasc.ZkController.checkChrootPath zkHost
> includes chroot
> [junit4] 2> 2539168 T14073
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
> [junit4] 2> 2539190 T14073 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
> [junit4] 2> 2539191 T14126 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@33c061dd
> name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2539191 T14073 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
> [junit4] 2> 2539192 T14073 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
> [junit4] 2> 2539209 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ConnectionManager.waitForConnected Waiting for client to connect to
> ZooKeeper
> [junit4] 2> 2539231 T14129 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@e61012e
> name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 2539231 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 2539235 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 2540240 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.ZkController.createEphemeralLiveNode Register node as live in
> ZooKeeper:/live_nodes/127.0.0.1:60967_emwi%2Fdt
> [junit4] 2> 2540241 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60967
> _emwi%2Fdt
> [junit4] 2> 2540244 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.Overseer.close Overseer (id=null) closing
> [junit4] 2> 2540246 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin
> used.
> [junit4] 2> 2540246 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't
> exist. Skipping setup for authorization module.
> [junit4] 2> 2540247 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CorePropertiesLocator.discover Looking for core definitions underneath
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores
> [junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {collection=collection1,
> schema=schema.xml,
> absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/,
> instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1, shard=, dataDir=data/,
> config=solrconfig.xml, transient=false, coreNodeName=, loadOnStartup=true,
> name=collection1}
> [junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CorePropertiesLocator.discoverUnder Found core collection1 in
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/
> [junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CorePropertiesLocator.discover Found 1 core definitions
> [junit4] 2> 2540267 T14130 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> x:collection1 oasc.ZkController.publish publishing core=collection1
> state=down collection=collection1
> [junit4] 2> 2540267 T14130 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> x:collection1 oasc.ZkController.publish numShards not found on descriptor -
> reading it from system property
> [junit4] 2> 2540268 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2540268 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 2540269 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "shard":null,
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
> [junit4] 2> "state":"down"} current state version: 4
> [junit4] 2> 2540269 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ReplicaMutator.updateState Update state numShards=3 message={
> [junit4] 2> "numShards":"3",
> [junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
> [junit4] 2> "operation":"state",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "shard":null,
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
> [junit4] 2> "state":"down"}
> [junit4] 2> 2540270 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ClusterStateMutator.createCollection building a new cName: collection1
> [junit4] 2> 2540270 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 2540372 T14112 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 2540372 T14129 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 2)
> [junit4] 2> 2540373 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 2)
> [junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.ZkController.waitForShardId waiting to find shard id in clusterstate
> for collection1
> [junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.ZkController.createCollectionZkNode Check for collection
> zkNode:collection1
> [junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 2541270 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ZkStateReader.readConfigName Load collection config
> from:/collections/collection1
> [junit4] 2> 2541271 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ZkStateReader.readConfigName path=/collections/collection1
> configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 2541271 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/'
> [junit4] 2> 2541291 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.Config.<init> loaded config solrconfig.xml with version 0
> [junit4] 2> 2541306 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
> [junit4] 2> 2541326 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
> [junit4] 2> 2541398 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 2541399 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.IndexSchema.readSchema Reading Solr Schema from
> /configs/conf1/schema.xml
> [junit4] 2> 2541414 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 2541635 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 2541637 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 2541638 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.FileExchangeRateProvider.reload Reloading exchange rates from file
> currency.xml
> [junit4] 2> 2541645 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oass.FileExchangeRateProvider.reload Reloading exchange rates from file
> currency.xml
> [junit4] 2> 2541682 T14130 n:127.0.0.1:60967_emwi%2Fdt
> oasc.CoreContainer.create Creating SolrCore 'collection1' using
> configuration from collection collection1
> [junit4] 2> 2541682 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at
> [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/], dataDir=[null]
> [junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans
> to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41aacb08
> [junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.CachingDirectoryFactory.get return new directory for
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data
> [junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.getNewIndexDir New index directory detected: old=null
> new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index/
> [junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.initIndex WARN [collection1] Solr index directory
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index' doesn't
> exist. Creating new index...
> [junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.CachingDirectoryFactory.get return new directory for
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index
> [junit4] 2> 2541685 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=18, maxMergeAtOnceExplicit=37,
> maxMergedSegmentMB=95.42578125, floorSegmentMB=0.2548828125,
> forceMergeDeletesPctAllowed=13.814119626493529, segmentsPerTier=18.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8506087588632382
> [junit4] 2> 2541708 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2>
> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
> [junit4] 2> 2541708 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 2541728 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "nodistrib"
> [junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "dedupe"
> [junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "distrib-dup-test-chain-explicit"
> [junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain
> defined as default, creating implicit default
> [junit4] 2> 2541733 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2541734 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2541736 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2541737 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 2541759 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.RequestHandlers.initHandlersFromConfig Registered paths:
> /admin/logging,/admin/threads,/admin/segments,/admin/ping,/admin/properties,/update/json,/update,standard,/config,/admin/file,/admin/system,/get,/replication,/admin/mbeans,/admin/plugins,/admin/luke,/update/json/docs,/update/csv,/schema
> [junit4] 2> 2541760 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.initStatsCache Using default statsCache cache:
> org.apache.solr.search.stats.LocalStatsCache
> [junit4] 2> 2541775 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.UpdateHandler.<init> Using UpdateLog implementation:
> org.apache.solr.update.UpdateLog
> [junit4] 2> 2541775 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
> numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 2541776 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 2541776 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 2541783 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class
> org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
> maxMergeAtOnce=31, maxMergeAtOnceExplicit=10,
> maxMergedSegmentMB=1.3017578125, floorSegmentMB=0.5830078125,
> forceMergeDeletesPctAllowed=8.067275561099606, segmentsPerTier=19.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6133962934564373
> [junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2>
> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest
> 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
> [junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oass.SolrIndexSearcher.<init> Opening Searcher@1ec1b8d5[collection1] main
> [junit4] 2> 2541785 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oascc.ZkStateReader.readConfigName Load collection config
> from:/collections/collection1
> [junit4] 2> 2541786 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oascc.ZkStateReader.readConfigName path=/collections/collection1
> configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 2541786 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage
> for the RestManager with znodeBase: /configs/conf1
> [junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
> [junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.RestManager.init Initializing RestManager with initArgs: {}
> [junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.load Reading _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
> [junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json
> using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasr.RestManager.init Initializing 0 registered ManagedResources
> [junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4] 2> 2541789 T14131 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.SolrCore.registerSearcher [collection1] Registered new searcher
> Searcher@1ec1b8d5[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> [junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max
> value of version field
> [junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of
> _version_ for 65536 version buckets from index
> [junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_,
> cannot seed version bucket highest value from index
> [junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in
> index or recent updates, using new clock 1502858774696689664
> [junit4] 2> 2541793 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version
> buckets with highest version 1502858774696689664
> [junit4] 2> 2541793 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
> [junit4] 2> 2541802 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1
> oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 2541822 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:60967/emwi/dt
> collection:collection1 shard:shard1
> [junit4] 2> 2541904 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oass.SolrDispatchFilter.init
> user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
> [junit4] 2> 2541904 T14073 n:127.0.0.1:60967_emwi%2Fdt
> oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 2541907 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 2541910 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess
> Running the leader process for shard shard1
> [junit4] 2> 2541911 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2541911 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I
> may be the new leader - try and sync
> [junit4] 2> ASYNC NEW_CORE C6969 name=collection1
> org.apache.solr.core.SolrCore@4112a586 url=
> http://127.0.0.1:60967/emwi/dt/collection1 node=127.0.0.1:60967_emwi%2Fdt
> C6969_STATE=coll:collection1 core:collection1 props:{base_url=
> http://127.0.0.1:60967/emwi/dt, core=collection1,
> node_name=127.0.0.1:60967_emwi%2Fdt, state=down}
> [junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 C6969 oasc.SyncStrategy.sync Sync replicas to
> http://127.0.0.1:60967/emwi/dt/collection1/
> [junit4] 2> 2541912 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1"} current state
> version: 5
> [junit4] 2> 2541913 T14112 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 C6969 oasc.SyncStrategy.syncReplicas Sync Success -
> now sync replicas to me
> [junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 C6969 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:60967/emwi/dt/collection1/ has no replicas
> [junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I
> am the new leader: http://127.0.0.1:60967/emwi/dt/collection1/ shard1
> [junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1
> s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
> [junit4] 2> 2541913 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 2)
> [junit4] 2> 2541915 T14129 n:127.0.0.1:60967_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 2)
> [junit4] 2> 2541919 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on
> path /overseer/queue state SyncConnected
> [junit4] 2> 2541920 T14102 n:127.0.0.1:43586_emwi%2Fdt
> oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message
> = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "state":"active"} current state version: 6
> [junit4] 2> 2542022 T14101 n:127.0.0.1:43586_emwi%2Fdt
> oascc.ZkStateReader$2.process A cluster state change: WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has
> occurred - updating... (live nodes size: 2)
> [junit4]
--
- Mark
about.me/markrmiller
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 8 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/8/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
Error Message:
shard3 is not consistent. Got 1164 from http://127.0.0.1:35113/emwi/dt/collection1lastClient and got 1020 from http://127.0.0.1:42407/emwi/dt/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 1164 from http://127.0.0.1:35113/emwi/dt/collection1lastClient and got 1020 from http://127.0.0.1:42407/emwi/dt/collection1
at __randomizedtesting.SeedInfo.seed([3E9C70166BCD829F:B6C84FCCC531EF67]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1293)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1272)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10742 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/init-core-data-001
[junit4] 2> 2536136 T14073 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 2536137 T14073 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /emwi/dt
[junit4] 2> 2536142 T14073 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 2536146 T14074 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2536146 T14074 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 2536246 T14073 oasc.ZkTestServer.run start zk server on port:47741
[junit4] 2> 2536246 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2536262 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2536273 T14081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c3c999a name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2536274 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2536274 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2536274 T14073 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2536277 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2536286 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2536296 T14084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2808fb2e name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2536296 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2536297 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2536297 T14073 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 2536299 T14073 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 2536376 T14073 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 2536377 T14073 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 2536378 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2536379 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 2536381 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2536381 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 2536383 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2536384 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2536385 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2536386 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 2536388 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2536388 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 2536390 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2536390 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 2536392 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2536392 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 2536394 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2536394 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 2536396 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2536396 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2536398 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2536398 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 2536399 T14073 oasc.AbstractZkTestCase.putConfig put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2536400 T14073 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 2536401 T14075 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14db3a1674a0001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2536777 T14073 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1
[junit4] 2> 2536796 T14073 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 2536982 T14073 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@a67b61d{/emwi/dt,null,AVAILABLE}
[junit4] 2> 2536984 T14073 oejs.AbstractConnector.doStart Started ServerConnector@1ee52f41{HTTP/1.1}{127.0.0.1:43586}
[junit4] 2> 2536984 T14073 oejs.Server.doStart Started @2538520ms
[junit4] 2> 2536985 T14073 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores, hostPort=43586, hostContext=/emwi/dt, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/tempDir-001/control/data}
[junit4] 2> 2536985 T14073 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 2536985 T14073 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/'
[junit4] 2> 2536999 T14073 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/solr.xml
[junit4] 2> 2537012 T14073 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores
[junit4] 2> 2537013 T14073 oasc.CoreContainer.<init> New CoreContainer 2139689457
[junit4] 2> 2537013 T14073 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/]
[junit4] 2> 2537013 T14073 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/lib
[junit4] 2> 2537013 T14073 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/lib).
[junit4] 2> 2537020 T14073 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 2537022 T14073 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2537023 T14073 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2537023 T14073 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2537023 T14073 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 2537023 T14073 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47741/solr
[junit4] 2> 2537023 T14073 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2537024 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2537025 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2537027 T14098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@89ad624 name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2537027 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2537027 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2537047 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2537056 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51f61ec4 name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2537056 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2537057 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 2537059 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 2537061 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 2537063 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 2537064 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 2537066 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2537068 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 2537069 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 2537070 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43586_emwi%2Fdt
[junit4] 2> 2537070 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43586_emwi%2Fdt
[junit4] 2> 2537072 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 2537073 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 2537074 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2537075 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43586_emwi%2Fdt
[junit4] 2> 2537076 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 2537077 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer.start Overseer (id=93928673048657923-127.0.0.1:43586_emwi%2Fdt-n_0000000000) starting
[junit4] 2> 2537078 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 2537083 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2537103 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 2537103 T14103 n:127.0.0.1:43586_emwi%2Fdt oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 2537114 T14073 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2537117 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 2537117 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 2537118 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores
[junit4] 2> 2537119 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {dataDir=data/, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/, transient=false, name=collection1, shard=, coreNodeName=, loadOnStartup=true, collection=control_collection, config=solrconfig.xml, schema=schema.xml, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1}
[junit4] 2> 2537119 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/
[junit4] 2> 2537120 T14073 n:127.0.0.1:43586_emwi%2Fdt oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 2537135 T14105 n:127.0.0.1:43586_emwi%2Fdt c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 2537135 T14105 n:127.0.0.1:43586_emwi%2Fdt c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2537136 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2537136 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2537137 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
[junit4] 2> "state":"down"} current state version: 0
[junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
[junit4] 2> "state":"down"}
[junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 2537138 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2537139 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538136 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2538137 T14105 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2538138 T14105 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2538138 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/'
[junit4] 2> 2538162 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 2538168 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 2538190 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 2538224 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2538226 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 2538231 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2538562 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2538564 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2538565 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2538578 T14105 n:127.0.0.1:43586_emwi%2Fdt oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2538595 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41aacb08
[junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data
[junit4] 2> 2538596 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data/index/
[junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data/index
[junit4] 2> 2538597 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=95.42578125, floorSegmentMB=0.2548828125, forceMergeDeletesPctAllowed=13.814119626493529, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8506087588632382
[junit4] 2> 2538613 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2538613 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2538617 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2538617 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2538618 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2538619 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2538619 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2538622 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2538623 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2538624 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2538625 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2538629 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/logging,/admin/threads,/admin/segments,/admin/ping,/admin/properties,/update/json,/update,standard,/config,/admin/file,/admin/system,/get,/replication,/admin/mbeans,/admin/plugins,/admin/luke,/update/json/docs,/update/csv,/schema
[junit4] 2> 2538630 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 2538631 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2538631 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2538632 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=1.3017578125, floorSegmentMB=0.5830078125, forceMergeDeletesPctAllowed=8.067275561099606, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6133962934564373
[junit4] 2> 2538633 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@60b77795[collection1] main
[junit4] 2> 2538634 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2538635 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2538636 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2538637 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2538637 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 2538638 T14106 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@60b77795[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 2538638 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502858771390529536
[junit4] 2> 2538640 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1502858771390529536
[junit4] 2> 2538641 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 2538642 T14105 n:127.0.0.1:43586_emwi%2Fdt x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2538643 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43586/emwi/dt collection:control_collection shard:shard1
[junit4] 2> 2538643 T14073 n:127.0.0.1:43586_emwi%2Fdt oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 2538644 T14073 n:127.0.0.1:43586_emwi%2Fdt oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2538645 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 2538645 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2538646 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2538648 T14112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1069b44e name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2538648 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2538648 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2538649 T14073 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2538654 T14073 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 2538661 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2538662 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2538663 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2538664 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 2538664 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C6968 name=collection1 org.apache.solr.core.SolrCore@4b88ed77 url=http://127.0.0.1:43586/emwi/dt/collection1 node=127.0.0.1:43586_emwi%2Fdt C6968_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:43586/emwi/dt, core=collection1, node_name=127.0.0.1:43586_emwi%2Fdt, state=down}
[junit4] 2> 2538664 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 C6968 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43586/emwi/dt/collection1/
[junit4] 2> 2538665 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 C6968 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2538665 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 C6968 oasc.SyncStrategy.syncToMe http://127.0.0.1:43586/emwi/dt/collection1/ has no replicas
[junit4] 2> 2538666 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43586/emwi/dt/collection1/ shard1
[junit4] 2> 2538666 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 2538666 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538667 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538675 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2538677 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 2
[junit4] 2> 2538780 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538780 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538855 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:43586/emwi/dt/collection1/ and leader is http://127.0.0.1:43586/emwi/dt/collection1/
[junit4] 2> 2538856 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43586/emwi/dt
[junit4] 2> 2538856 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2538857 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 2538857 T14109 n:127.0.0.1:43586_emwi%2Fdt c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2538862 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2538864 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
[junit4] 2> "state":"active"} current state version: 3
[junit4] 2> 2538865 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:43586/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43586_emwi%2Fdt",
[junit4] 2> "state":"active"}
[junit4] 2> 2538969 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2538969 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2539067 T14073 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1
[junit4] 2> 2539068 T14073 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001
[junit4] 2> 2539069 T14073 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 2539089 T14073 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@457c1274{/emwi/dt,null,AVAILABLE}
[junit4] 2> 2539090 T14073 oejs.AbstractConnector.doStart Started ServerConnector@3c1774f0{HTTP/1.1}{127.0.0.1:60967}
[junit4] 2> 2539090 T14073 oejs.Server.doStart Started @2540626ms
[junit4] 2> 2539090 T14073 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/emwi/dt, hostPort=60967, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/tempDir-001/jetty1, solrconfig=solrconfig.xml}
[junit4] 2> 2539091 T14073 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 2539091 T14073 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/'
[junit4] 2> 2539115 T14073 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/solr.xml
[junit4] 2> 2539148 T14073 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores
[junit4] 2> 2539149 T14073 oasc.CoreContainer.<init> New CoreContainer 1270053320
[junit4] 2> 2539149 T14073 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/]
[junit4] 2> 2539149 T14073 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/lib
[junit4] 2> 2539149 T14073 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/lib).
[junit4] 2> 2539165 T14073 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 2539166 T14073 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2539167 T14073 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2539167 T14073 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2539167 T14073 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 2539167 T14073 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47741/solr
[junit4] 2> 2539168 T14073 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2539168 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2539190 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2539191 T14126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33c061dd name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2539191 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2539192 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2539209 T14073 n:127.0.0.1:60967_emwi%2Fdt oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2539231 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e61012e name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2539231 T14073 n:127.0.0.1:60967_emwi%2Fdt oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2539235 T14073 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2540240 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60967_emwi%2Fdt
[junit4] 2> 2540241 T14073 n:127.0.0.1:60967_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60967_emwi%2Fdt
[junit4] 2> 2540244 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2540246 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 2540246 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 2540247 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores
[junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {collection=collection1, schema=schema.xml, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1, shard=, dataDir=data/, config=solrconfig.xml, transient=false, coreNodeName=, loadOnStartup=true, name=collection1}
[junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/
[junit4] 2> 2540248 T14073 n:127.0.0.1:60967_emwi%2Fdt oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 2540267 T14130 n:127.0.0.1:60967_emwi%2Fdt c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2540267 T14130 n:127.0.0.1:60967_emwi%2Fdt c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2540268 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2540268 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2540269 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
[junit4] 2> "state":"down"} current state version: 4
[junit4] 2> 2540269 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
[junit4] 2> "state":"down"}
[junit4] 2> 2540270 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 2540270 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2540372 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2540372 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2540373 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 2541269 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2541270 T14130 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2541271 T14130 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2541271 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/'
[junit4] 2> 2541291 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 2541306 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 2541326 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.2.0
[junit4] 2> 2541398 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2541399 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 2541414 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2541635 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2541637 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2541638 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2541645 T14130 n:127.0.0.1:60967_emwi%2Fdt oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2541682 T14130 n:127.0.0.1:60967_emwi%2Fdt oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 2541682 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41aacb08
[junit4] 2> 2541683 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data
[junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 2541684 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.CachingDirectoryFactory.get return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 2541685 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=95.42578125, floorSegmentMB=0.2548828125, forceMergeDeletesPctAllowed=13.814119626493529, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8506087588632382
[junit4] 2> 2541708 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2541708 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2541728 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2541729 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2541730 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2541733 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2541734 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2541736 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2541737 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2541759 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/logging,/admin/threads,/admin/segments,/admin/ping,/admin/properties,/update/json,/update,standard,/config,/admin/file,/admin/system,/get,/replication,/admin/mbeans,/admin/plugins,/admin/luke,/update/json/docs,/update/csv,/schema
[junit4] 2> 2541760 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 2541775 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2541775 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2541776 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2541776 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2541783 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=1.3017578125, floorSegmentMB=0.5830078125, forceMergeDeletesPctAllowed=8.067275561099606, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6133962934564373
[junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-1-001/cores/collection1/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2541784 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@1ec1b8d5[collection1] main
[junit4] 2> 2541785 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2541786 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2541786 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2541787 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2541788 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 2541789 T14131 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ec1b8d5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 2541791 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in index or recent updates, using new clock 1502858774696689664
[junit4] 2> 2541793 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1502858774696689664
[junit4] 2> 2541793 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 2541802 T14130 n:127.0.0.1:60967_emwi%2Fdt x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2541822 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60967/emwi/dt collection:collection1 shard:shard1
[junit4] 2> 2541904 T14073 n:127.0.0.1:60967_emwi%2Fdt oass.SolrDispatchFilter.init user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1
[junit4] 2> 2541904 T14073 n:127.0.0.1:60967_emwi%2Fdt oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2541907 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 2541910 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2541911 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2541911 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C6969 name=collection1 org.apache.solr.core.SolrCore@4112a586 url=http://127.0.0.1:60967/emwi/dt/collection1 node=127.0.0.1:60967_emwi%2Fdt C6969_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60967/emwi/dt, core=collection1, node_name=127.0.0.1:60967_emwi%2Fdt, state=down}
[junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 C6969 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60967/emwi/dt/collection1/
[junit4] 2> 2541912 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1"} current state version: 5
[junit4] 2> 2541913 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2541912 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 C6969 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 C6969 oasc.SyncStrategy.syncToMe http://127.0.0.1:60967/emwi/dt/collection1/ has no replicas
[junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60967/emwi/dt/collection1/ shard1
[junit4] 2> 2541915 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 2541913 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2541915 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2541919 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2541920 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 6
[junit4] 2> 2542022 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542022 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542022 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542070 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:60967/emwi/dt/collection1/ and leader is http://127.0.0.1:60967/emwi/dt/collection1/
[junit4] 2> 2542070 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60967/emwi/dt
[junit4] 2> 2542070 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2542070 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 2542071 T14134 n:127.0.0.1:60967_emwi%2Fdt c:collection1 s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2542072 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2542072 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
[junit4] 2> "state":"active"} current state version: 7
[junit4] 2> 2542073 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:60967/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60967_emwi%2Fdt",
[junit4] 2> "state":"active"}
[junit4] 2> 2542175 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542175 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542176 T14101 n:127.0.0.1:43586_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2542237 T14073 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores/collection1
[junit4] 2> 2542238 T14073 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001
[junit4] 2> 2542239 T14073 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 2542304 T14073 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@636e36d4{/emwi/dt,null,AVAILABLE}
[junit4] 2> 2542305 T14073 oejs.AbstractConnector.doStart Started ServerConnector@58d1cb4b{HTTP/1.1}{127.0.0.1:52805}
[junit4] 2> 2542305 T14073 oejs.Server.doStart Started @2543841ms
[junit4] 2> 2542306 T14073 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostPort=52805, hostContext=/emwi/dt, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores, solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/tempDir-001/jetty2, solrconfig=solrconfig.xml}
[junit4] 2> 2542306 T14073 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@72b5615d
[junit4] 2> 2542306 T14073 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/'
[junit4] 2> 2542333 T14073 oasc.SolrXmlConfig.fromFile Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/solr.xml
[junit4] 2> 2542359 T14073 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores
[junit4] 2> 2542359 T14073 oasc.CoreContainer.<init> New CoreContainer 1654925878
[junit4] 2> 2542359 T14073 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/]
[junit4] 2> 2542360 T14073 oasc.CoreContainer.load loading shared library: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/lib
[junit4] 2> 2542360 T14073 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/lib).
[junit4] 2> 2542380 T14073 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 2542382 T14073 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 2542382 T14073 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2542382 T14073 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2542383 T14073 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 2542383 T14073 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47741/solr
[junit4] 2> 2542383 T14073 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2542383 T14073 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2542402 T14073 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2542423 T14148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@212c0d11 name:ZooKeeperConnection Watcher:127.0.0.1:47741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2542423 T14073 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2542423 T14073 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2542454 T14073 n:127.0.0.1:52805_emwi%2Fdt oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2542463 T14151 n:127.0.0.1:52805_emwi%2Fdt oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21bff0ee name:ZooKeeperConnection Watcher:127.0.0.1:47741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2542463 T14073 n:127.0.0.1:52805_emwi%2Fdt oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2542467 T14073 n:127.0.0.1:52805_emwi%2Fdt oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2543471 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52805_emwi%2Fdt
[junit4] 2> 2543472 T14073 n:127.0.0.1:52805_emwi%2Fdt oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52805_emwi%2Fdt
[junit4] 2> 2543475 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2543476 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 2543476 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 2543477 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CorePropertiesLocator.discover Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores
[junit4] 2> 2543478 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {collection=collection1, transient=false, dataDir=data/, name=collection1, shard=, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, coreNodeName=, absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores/collection1/, instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores/collection1}
[junit4] 2> 2543479 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest 3E9C70166BCD829F-001/shard-2-001/cores/collection1/
[junit4] 2> 2543479 T14073 n:127.0.0.1:52805_emwi%2Fdt oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 2543491 T14152 n:127.0.0.1:52805_emwi%2Fdt c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2543491 T14152 n:127.0.0.1:52805_emwi%2Fdt c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2543492 T14101 n:127.0.0.1:43586_emwi%2Fdt oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 2543492 T14152 n:127.0.0.1:52805_emwi%2Fdt oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2543493 T14102 n:127.0.0.1:43586_emwi%2Fdt oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:52805/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:52805_emwi%2Fdt",
[junit4] 2> "state":"down"} current state version: 8
[junit4] 2> 2543493 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1:52805/emwi/dt",
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:52805_emwi%2Fdt",
[junit4] 2> "state":"down"}
[junit4] 2> 2543494 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Collection already exists with numShards=3
[junit4] 2> 2543494 T14102 n:127.0.0.1:43586_emwi%2Fdt oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 2543597 T14112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2543597 T14129 n:127.0.0.1:60967_emwi%2Fdt oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChang
[...truncated too long message...]
/solr/solrj/lib/commons-io-2.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpclient-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpcore-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpmime-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/noggit-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/slf4j-api-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/stax2-api-3.1.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/zookeeper-3.4.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/javax.servlet-api-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-continuation-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-deploy-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-http-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-io-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-jmx-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-rewrite-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-security-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-server-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-servlet-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-servlets-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-util-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-webapp-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-xml-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/apacheds-all-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/commons-collections-3.2.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/analysis/icu/lucene-analyzers-icu-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J0-20150602_045431_184.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J0-20150602_045431_184.suites
[junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
Total time: 172 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 1416694303 bytes
Compression is 0.0%
Took 4 min 29 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 7 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/7/
All tests passed
Build Log:
[...truncated 10373 lines...]
[junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150531_005833_013.sysout
[junit4] >>> JVM J1: stdout (verbatim) ----
[junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps/java_pid25005.hprof ...
[junit4] Heap dump file created [602641422 bytes in 6.781 secs]
[junit4] <<< JVM J1: EOF ----
[junit4] JVM J1: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150531_005833_013.syserr
[junit4] >>> JVM J1: stderr (verbatim) ----
[junit4] WARN: Unhandled exception in event serialization. -> java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] at java.nio.CharBuffer.wrap(CharBuffer.java:369)
[junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:265)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
[junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.string(JsonWriter.java:535)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:364)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:626)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapters$22.write(TypeAdapters.java:578)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.Streams.write(Streams.java:67)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.GsonToMiniGsonTypeAdapterFactory$3.write(GsonToMiniGsonTypeAdapterFactory.java:98)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:66)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:82)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:194)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:512)
[junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
[junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
[junit4] at java.io.PrintStream.flush(PrintStream.java:338)
[junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
[junit4] at java.io.PrintStream.write(PrintStream.java:482)
[junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
[junit4] at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
[junit4] at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
[junit4] at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
[junit4] at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
[junit4] at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
[junit4] at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
[junit4] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
[junit4] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
[junit4] <<< JVM J1: EOF ----
[...truncated 959 lines...]
[junit4] ERROR: JVM J1 ended with an exception, command line: /x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=1695EBC35849A6BC -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=5.2.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=5.2.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/test-framework/lib/junit4-ant-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/src/test-files:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-solrj/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/analysis/common/lucene-analyzers-common-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/codecs/lucene-codecs-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/backward-codecs/lucene-backward-codecs-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/highlighter/lucene-highlighter-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/memory/lucene-memory-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/misc/lucene-misc-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/spatial/lucene-spatial-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/expressions/lucene-expressions-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/suggest/lucene-suggest-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/grouping/lucene-grouping-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/queries/lucene-queries-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/queryparser/lucene-queryparser-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/join/lucene-join-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/antlr-runtime-3.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/asm-4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/asm-commons-4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-cli-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-codec-1.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-collections-3.2.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-configuration-1.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-fileupload-1.2.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/commons-lang-2.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/dom4j-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/fastutil-6.5.11.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/guava-14.0.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hadoop-annotations-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hadoop-auth-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hadoop-common-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hadoop-hdfs-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hll-1.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/hppc-0.5.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/htrace-core-3.0.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/joda-time-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/log4j-1.2.17.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/org.restlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/protobuf-java-2.5.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/slf4j-log4j12-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/spatial4j-0.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/lib/t-digest-3.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/commons-io-2.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpclient-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpcore-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/httpmime-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/noggit-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/slf4j-api-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/stax2-api-3.1.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/solrj/lib/zookeeper-3.4.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/javax.servlet-api-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-continuation-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-deploy-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-http-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-io-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-jmx-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-rewrite-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-security-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-server-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-servlet-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-servlets-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-util-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-webapp-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/server/lib/jetty-xml-9.2.10.v20150310.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/apacheds-all-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/commons-collections-3.2.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/build/analysis/icu/lucene-analyzers-icu-5.2.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.7/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150531_005833_013.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150531_005833_013.suites
[junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
Total time: 186 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 1600050369 bytes
Compression is 0.0%
Took 4 min 46 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 6 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/6/
4 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=7854, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=7854, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:35784, http://127.0.0.1:46980, http://127.0.0.1:50962, http://127.0.0.1:37941, http://127.0.0.1:52086]
at __randomizedtesting.SeedInfo.seed([A64FFB27BB2DE866]:0)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:887)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:35784, http://127.0.0.1:46980, http://127.0.0.1:50962, http://127.0.0.1:37941, http://127.0.0.1:52086]
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1621)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1642)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:877)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:46980: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qn-
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
... 7 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest
Error Message:
ERROR: SolrIndexSearcher opens=497 closes=382
Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=497 closes=382
at __randomizedtesting.SeedInfo.seed([A64FFB27BB2DE866]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:496)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:232)
at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest
Error Message:
201 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 1) Thread[id=9875, name=searcherExecutor-4553-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=9766, name=searcherExecutor-4380-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=7844, name=searcherExecutor-2736-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=7744, name=searcherExecutor-2648-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=10567, name=coreZkRegister-2526-thread-10-processing-{node_name=127.0.0.1:37941_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:682) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:651) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:964) at org.apache.solr.cloud.ZkController.register(ZkController.java:901) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 6) Thread[id=9759, name=searcherExecutor-4370-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=10098, name=searcherExecutor-4728-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=10333, name=searcherExecutor-4744-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=9636, name=qtp103799490-9636, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=7528, name=Thread-3112, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1037) 11) Thread[id=7591, name=Scheduler-349965222, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=10033, name=searcherExecutor-4679-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 13) Thread[id=9855, name=searcherExecutor-4536-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=7736, name=searcherExecutor-2629-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=10604, name=searcherExecutor-4804-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16) Thread[id=9785, name=searcherExecutor-4427-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=7516, name=org.eclipse.jetty.server.session.HashSessionManager@66bcfa3dTimer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=7539, name=qtp106457986-7539-selector-ServerConnectorManager@763ace0a/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=11020, name=qtp1132163769-11020, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 20) Thread[id=10597, name=searcherExecutor-4795-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 21) Thread[id=7635, name=Scheduler-879719143, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 22) Thread[id=9774, name=searcherExecutor-4384-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 23) Thread[id=9655, name=searcherExecutor-4234-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 24) Thread[id=7645, name=searcherExecutor-2551-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=10394, name=searcherExecutor-4760-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 26) Thread[id=7625, name=searcherExecutor-2531-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 27) Thread[id=7541, name=qtp106457986-7541-acceptor-0@ead2f2a-ServerConnector@2a311af5{HTTP/1.1}{127.0.0.1:52086}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 28) Thread[id=9882, name=searcherExecutor-4545-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 29) Thread[id=10601, name=searcherExecutor-4793-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 30) Thread[id=7617, name=org.eclipse.jetty.server.session.HashSessionManager@1059e659Timer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 31) Thread[id=10593, name=searcherExecutor-4785-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 32) Thread[id=7579, name=Thread-3134, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1037) 33) Thread[id=10031, name=searcherExecutor-4674-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=7570, name=org.eclipse.jetty.server.session.HashSessionManager@40c8ab8bTimer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 35) Thread[id=7610, name=qtp374555500-7610-acceptor-0@7110e9f7-ServerConnector@6102562e{HTTP/1.1}{127.0.0.1:37941}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 36) Thread[id=10321, name=zkCallback-890-thread-7-processing-{node_name=127.0.0.1:50962_}-SendThread(127.0.0.1:35427), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 37) Thread[id=10392, name=searcherExecutor-4751-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 38) Thread[id=10331, name=zkCallback-887-thread-17-processing-{node_name=127.0.0.1:46980_}-SendThread(127.0.0.1:35427), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 39) Thread[id=10603, name=searcherExecutor-4799-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 40) Thread[id=10043, name=searcherExecutor-4690-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=7515, name=Scheduler-946984731, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 42) Thread[id=9921, name=searcherExecutor-4609-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 43) Thread[id=7601, name=Thread-3145, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1037) 44) Thread[id=9812, name=searcherExecutor-4460-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 45) Thread[id=10599, name=searcherExecutor-4801-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 46) Thread[id=7561, name=qtp1644866606-7561-selector-ServerConnectorManager@ecaa3ec/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 47) Thread[id=7641, name=searcherExecutor-2541-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 48) Thread[id=9956, name=searcherExecutor-4620-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 49) Thread[id=7730, name=searcherExecutor-2616-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 50) Thread[id=9958, name=searcherExecutor-4621-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 51) Thread[id=7509, name=qtp103799490-7509-selector-ServerConnectorManager@787956c7/1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 52) Thread[id=8175, name=qtp103799490-8175, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 53) Thread[id=9966, name=searcherExecutor-4625-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 54) Thread[id=9757, name=searcherExecutor-4395-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 55) Thread[id=9805, name=searcherExecutor-4486-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 56) Thread[id=9031, name=qtp1644866606-9031, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 57) Thread[id=11008, name=qtp1132163769-11008, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 58) Thread[id=9817, name=searcherExecutor-4479-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 59) Thread[id=11013, name=qtp106457986-11013, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 60) Thread[id=10600, name=searcherExecutor-4798-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 61) Thread[id=9722, name=searcherExecutor-4324-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 62) Thread[id=9736, name=searcherExecutor-4325-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 63) Thread[id=7547, name=Scheduler-392807753, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 64) Thread[id=9755, name=searcherExecutor-4350-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 65) Thread[id=10091, name=qtp1644866606-10091, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 66) Thread[id=10022, name=searcherExecutor-4675-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 67) Thread[id=10003, name=searcherExecutor-4646-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 68) Thread[id=10335, name=zkCallback-878-thread-19-processing-{node_name=127.0.0.1:35784_}-SendThread(127.0.0.1:35427), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 69) Thread[id=7750, name=searcherExecutor-2642-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 70) Thread[id=11034, name=qtp106457986-11034, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 71) Thread[id=7731, name=searcherExecutor-2617-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 72) Thread[id=7609, name=qtp374555500-7609-selector-ServerConnectorManager@4021a5a6/1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 73) Thread[id=10869, name=coreZkRegister-2485-thread-8-processing-{node_name=127.0.0.1:35784_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:682) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:651) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:964) at org.apache.solr.cloud.ZkController.register(ZkController.java:901) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 74) Thread[id=7825, name=searcherExecutor-2724-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 75) Thread[id=11030, name=qtp1132163769-11030, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 76) Thread[id=9756, name=searcherExecutor-4377-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 77) Thread[id=7507, name=qtp103799490-7507-selector-ServerConnectorManager@787956c7/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 78) Thread[id=9763, name=searcherExecutor-4364-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 79) Thread[id=9769, name=searcherExecutor-4398-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 80) Thread[id=10047, name=searcherExecutor-4689-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 81) Thread[id=10354, name=OverseerCollectionProcessor-93904249330466891-127.0.0.1:52086_-n_0000000005, state=TIMED_WAITING, group=Overseer collection creation process.] at java.lang.Object.wait(Native Method) at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320) at org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:447) at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:301) at java.lang.Thread.run(Thread.java:745) 82) Thread[id=7642, name=searcherExecutor-2542-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 83) Thread[id=10001, name=searcherExecutor-4657-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 84) Thread[id=10099, name=searcherExecutor-4723-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 85) Thread[id=11025, name=qtp106457986-11025, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) 86) Thread[id=10564, name=coreZkRegister-2526-thread-7-processing-{node_name=127.0.0.1:37941_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:682) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:651) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:964) at org.apache.solr.cloud.ZkController.register(ZkController.java:901) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 87) Thread[id=10595, name=searcherExecutor-4790-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 88) Thread[id=7616, name=Scheduler-1494197920, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 89) Thread[id=9721, name=searcherExecutor-4303-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 90) Thread[id=9876, name=searcherExecutor-4550-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 91) Thread[id=10323, name=zkCallback-884-thread-16-processing-{node_name=127.0.0.1:52086_}-SendThread(127.0.0.1:35427), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 92) Thread[id=9718, name=searcherExecutor-4296-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 93) Thread[id=10395, name=searcherExecutor-4761-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 94) Thread[id=9890, name=searcherExecutor-4582-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 95) Thread[id=7741, name=searcherExecutor-2640-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 96) Thread[id=7583, name=qtp1132163769-7583-selector-ServerConnectorManager@57d565b5/0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 97) Thread[id=9735, name=searcherExecutor-4300-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 98) Thread[id=10332, name=zkCallback-887-thread-17-processing-{node_name=127.0.0.1:46980_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 99) Thread[id=7746, name=searcherExecutor-2641-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 100) Thread[id=9764, name=searcherExecutor-4381-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 101) Thread[id=7563, name=qtp1644866606-7563-acceptor-0@19cfe28a-ServerConnector@173b0f75{HTTP/1.1}{127.0.0.1:46980}, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) 102) Thread[id=10000, name=searcherExecutor-4659-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 103) Thread[id=7556, name=searcherExecutor-2501-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 104) Thread[id=7548, name=org.eclipse.jetty.server.session.HashSessionManager@5df3d529Timer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 105) Thread[id=9753, name=searcherExecutor-4348-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 106) Thread[id=10596, name=searcherExecutor-4791-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 107) Thread[id=11027, name=qtp374555500-11027, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.
[...truncated too long message...]
unit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 147) Thread[id=9891, name=searcherExecutor-4574-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 148) Thread[id=7527, name=searcherExecutor-2491-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 149) Thread[id=7548, name=org.eclipse.jetty.server.session.HashSessionManager@5df3d529Timer, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 150) Thread[id=7591, name=Scheduler-349965222, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 151) Thread[id=9753, name=searcherExecutor-4348-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 152) Thread[id=10033, name=searcherExecutor-4679-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 153) Thread[id=10596, name=searcherExecutor-4791-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 154) Thread[id=9855, name=searcherExecutor-4536-thread-1, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([A64FFB27BB2DE866]:0)
[junit4] Completed [213/497] on J1 in 1015.61s, 1 test, 1 failure, 3 errors <<< FAILURES!
[...truncated 860 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1433: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 497 suites, 1974 tests, 3 suite-level errors, 1 error, 1084 ignored (6 assumptions)
Total time: 154 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 671851442 bytes
Compression is 0.0%
Took 1 min 48 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-5.2 - Build # 5 - Still Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.2/5/
4 tests failed.
REGRESSION: org.apache.solr.cloud.MultiThreadedOCPTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=6237, name=parallelCoreAdminExecutor-2470-thread-14, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6237, name=parallelCoreAdminExecutor-2470-thread-14, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
at __randomizedtesting.SeedInfo.seed([1E0D9EBC8B4DDDE1:9659A16625B1B019]:0)
Caused by: java.lang.AssertionError: Too many closes on SolrCore
at __randomizedtesting.SeedInfo.seed([1E0D9EBC8B4DDDE1]:0)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1157)
at org.apache.solr.common.util.IOUtils.closeQuietly(IOUtils.java:31)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:652)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
at org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1249)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.search.TestMaxScoreQueryParser
Error Message:
Stack Trace:
java.lang.AssertionError
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:113)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.search.TestMaxScoreQueryParser
Error Message:
45 threads leaked from SUITE scope at org.apache.solr.search.TestMaxScoreQueryParser: 1) Thread[id=15807, name=coreZkRegister-4049-thread-71-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=15832, name=coreZkRegister-4049-thread-96-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=15823, name=coreZkRegister-4049-thread-87-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=15827, name=coreZkRegister-4049-thread-91-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=15806, name=coreZkRegister-4049-thread-70-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 6) Thread[id=15796, name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 7) Thread[id=15816, name=coreZkRegister-4049-thread-80-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=15841, name=zkCallback-1207-thread-99, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=15794, name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 10) Thread[id=15802, name=OverseerCollectionProcessor-93898657861468232-127.0.0.1:57064_-n_0000000021, state=TIMED_WAITING, group=Overseer collection creation process.] at java.lang.Object.wait(Native Method) at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320) at org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:447) at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:301) at java.lang.Thread.run(Thread.java:745) 11) Thread[id=15820, name=coreZkRegister-4049-thread-84-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=15791, name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-SendThread(127.0.0.1:55012), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 13) Thread[id=15834, name=coreZkRegister-4049-thread-98, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=15836, name=coreZkRegister-4049-thread-100, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=15808, name=coreZkRegister-4049-thread-72-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16) Thread[id=15814, name=coreZkRegister-4049-thread-78-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=15801, name=OverseerStateUpdate-93898657861468232-127.0.0.1:57064_-n_0000000021, state=WAITING, group=Overseer state updater.] at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:528) at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:269) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:152) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=15830, name=coreZkRegister-4049-thread-94, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=15803, name=OverseerHdfsCoreFailoverThread-93898657861468232-127.0.0.1:57064_-n_0000000021, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139) at java.lang.Thread.run(Thread.java:745) 20) Thread[id=15828, name=coreZkRegister-4049-thread-92-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 21) Thread[id=15812, name=coreZkRegister-4049-thread-76-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 22) Thread[id=15792, name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 23) Thread[id=15810, name=coreZkRegister-4049-thread-74-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 24) Thread[id=15821, name=coreZkRegister-4049-thread-85-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=15819, name=coreZkRegister-4049-thread-83-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 26) Thread[id=15833, name=coreZkRegister-4049-thread-97, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 27) Thread[id=15824, name=coreZkRegister-4049-thread-88-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 28) Thread[id=15795, name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-SendThread(127.0.0.1:55012), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 29) Thread[id=15811, name=coreZkRegister-4049-thread-75-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 30) Thread[id=15835, name=coreZkRegister-4049-thread-99, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 31) Thread[id=15804, name=coreZkRegister-4049-thread-68-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 32) Thread[id=15829, name=coreZkRegister-4049-thread-93-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 33) Thread[id=15840, name=zkCallback-1207-thread-98, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=15817, name=coreZkRegister-4049-thread-81-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 35) Thread[id=15822, name=coreZkRegister-4049-thread-86-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 36) Thread[id=15805, name=coreZkRegister-4049-thread-69-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 37) Thread[id=15809, name=coreZkRegister-4049-thread-73-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:682) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:651) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:964) at org.apache.solr.cloud.ZkController.register(ZkController.java:901) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 38) Thread[id=15793, name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-SendThread(127.0.0.1:55012), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 39) Thread[id=15825, name=coreZkRegister-4049-thread-89-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 40) Thread[id=15831, name=coreZkRegister-4049-thread-95-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=15818, name=coreZkRegister-4049-thread-82-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 42) Thread[id=15815, name=coreZkRegister-4049-thread-79-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 43) Thread[id=15813, name=coreZkRegister-4049-thread-77-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 44) Thread[id=15839, name=zkCallback-1207-thread-97, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 45) Thread[id=15826, name=coreZkRegister-4049-thread-90-processing-{node_name=127.0.0.1:57064_}, state=BLOCKED, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472) at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348) at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075) at org.apache.solr.cloud.ZkController.register(ZkController.java:888) at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 45 threads leaked from SUITE scope at org.apache.solr.search.TestMaxScoreQueryParser:
1) Thread[id=15807, name=coreZkRegister-4049-thread-71-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=15832, name=coreZkRegister-4049-thread-96-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=15823, name=coreZkRegister-4049-thread-87-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=15827, name=coreZkRegister-4049-thread-91-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=15806, name=coreZkRegister-4049-thread-70-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
6) Thread[id=15796, name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
7) Thread[id=15816, name=coreZkRegister-4049-thread-80-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
8) Thread[id=15841, name=zkCallback-1207-thread-99, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
9) Thread[id=15794, name=zkCallback-1217-thread-29-processing-{node_name=127.0.0.1:38044_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
10) Thread[id=15802, name=OverseerCollectionProcessor-93898657861468232-127.0.0.1:57064_-n_0000000021, state=TIMED_WAITING, group=Overseer collection creation process.]
at java.lang.Object.wait(Native Method)
at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:276)
at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:320)
at org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:447)
at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:301)
at java.lang.Thread.run(Thread.java:745)
11) Thread[id=15820, name=coreZkRegister-4049-thread-84-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
12) Thread[id=15791, name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-SendThread(127.0.0.1:55012), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
13) Thread[id=15834, name=coreZkRegister-4049-thread-98, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14) Thread[id=15836, name=coreZkRegister-4049-thread-100, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15) Thread[id=15808, name=coreZkRegister-4049-thread-72-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16) Thread[id=15814, name=coreZkRegister-4049-thread-78-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
17) Thread[id=15801, name=OverseerStateUpdate-93898657861468232-127.0.0.1:57064_-n_0000000021, state=WAITING, group=Overseer state updater.]
at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:528)
at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:269)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:152)
at java.lang.Thread.run(Thread.java:745)
18) Thread[id=15830, name=coreZkRegister-4049-thread-94, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
19) Thread[id=15803, name=OverseerHdfsCoreFailoverThread-93898657861468232-127.0.0.1:57064_-n_0000000021, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
at java.lang.Thread.run(Thread.java:745)
20) Thread[id=15828, name=coreZkRegister-4049-thread-92-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
21) Thread[id=15812, name=coreZkRegister-4049-thread-76-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
22) Thread[id=15792, name=zkCallback-1210-thread-22-processing-{node_name=127.0.0.1:39037_}-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
23) Thread[id=15810, name=coreZkRegister-4049-thread-74-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
24) Thread[id=15821, name=coreZkRegister-4049-thread-85-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
25) Thread[id=15819, name=coreZkRegister-4049-thread-83-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
26) Thread[id=15833, name=coreZkRegister-4049-thread-97, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
27) Thread[id=15824, name=coreZkRegister-4049-thread-88-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
28) Thread[id=15795, name=zkCallback-1213-thread-28-processing-{node_name=127.0.0.1:53961_}-SendThread(127.0.0.1:55012), state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
29) Thread[id=15811, name=coreZkRegister-4049-thread-75-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
30) Thread[id=15835, name=coreZkRegister-4049-thread-99, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
31) Thread[id=15804, name=coreZkRegister-4049-thread-68-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
32) Thread[id=15829, name=coreZkRegister-4049-thread-93-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
33) Thread[id=15840, name=zkCallback-1207-thread-98, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
34) Thread[id=15817, name=coreZkRegister-4049-thread-81-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
35) Thread[id=15822, name=coreZkRegister-4049-thread-86-processing-{node_name=127.0.0.1:57064_}, state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:472)
at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:234)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1075)
at org.apache.solr.cloud.ZkController.register(ZkController.java:888)
at org.apache.solr.cloud.ZkControl
[...truncated too long message...]
java:603)
[junit4] 2> at java.io.PrintWriter.println(PrintWriter.java:756)
[junit4] 2> at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
[junit4] 2> at java.lang.Throwable.printStackTrace(Throwable.java:658)
[junit4] 2> at java.lang.Throwable.printStackTrace(Throwable.java:721)
[junit4] 2> at org.apache.solr.common.SolrException.toStr(SolrException.java:170)
[junit4] 2> at org.apache.solr.common.SolrException.log(SolrException.java:133)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:164)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2>
[junit4] 2> 10154996 T15161 n:127.0.0.1:57064_ oejs.ServletHandler.doHandle WARN Error for /admin/cores java.lang.OutOfMemoryError: Java heap space
[junit4] 2> at java.util.Arrays.copyOf(Arrays.java:2367)
[junit4] 2> at java.io.CharArrayWriter.write(CharArrayWriter.java:122)
[junit4] 2> at java.io.PrintWriter.write(PrintWriter.java:456)
[junit4] 2> at java.io.PrintWriter.write(PrintWriter.java:473)
[junit4] 2> at java.io.PrintWriter.print(PrintWriter.java:603)
[junit4] 2> at java.io.PrintWriter.println(PrintWriter.java:756)
[junit4] 2> at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
[junit4] 2> at java.lang.Throwable.printStackTrace(Throwable.java:658)
[junit4] 2> at java.lang.Throwable.printStackTrace(Throwable.java:721)
[junit4] 2> at org.apache.solr.common.SolrException.toStr(SolrException.java:170)
[junit4] 2> at org.apache.solr.common.SolrException.log(SolrException.java:133)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:164)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:646)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:417)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2>
[junit4] May 28, 2015 12:31:23 AM com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler handleException
[junit4] SEVERE: Could not dispatch event: com.carrotsearch.ant.tasks.junit4.listeners.TextReport@105cf2a8 to public void com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
[junit4] May 28, 2015 12:31:23 AM com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler handleException
[junit4] SEVERE: Could not dispatch event: com.carrotsearch.ant.tasks.junit4.listeners.TextReport@22ae4c81 to public void com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
[junit4] May 28, 2015 12:31:23 AM com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.eventbus.EventBus$LoggingSubscriberExceptionHandler handleException
[junit4] SEVERE: Could not dispatch event: com.carrotsearch.ant.tasks.junit4.listeners.TextReport@5139b168 to public void com.carrotsearch.ant.tasks.junit4.listeners.TextReport.onSuiteResult(com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent)
[junit4] Suite: org.apache.solr.cloud.VMParamsZkACLAndCredentialsProvidersTest
[junit4] Completed [496/497] on J1 in 0.93s, 4 tests, 4 skipped
[junit4]
[junit4] Suite: org.apache.solr.search.TestCustomSort
[junit4] Completed [497/497] on J1 in 0.01s, 1 test, 1 skipped
[junit4]
[junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build/solr-core/test/temp/junit4-J1-20150527_214030_137.sysout
[junit4] >>> JVM J1: stdout (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/heapdumps/java_pid1911.hprof ...
[junit4] Heap dump file created [646867182 bytes in 6.369 secs]
[junit4] <<< JVM J1: EOF ----
[junit4]
[junit4] Tests with failures:
[junit4] - org.apache.solr.cloud.MultiThreadedOCPTest.test
[junit4] - org.apache.solr.cloud.CollectionsAPIDistributedZkTest (suite)
[junit4]
[junit4]
[junit4] JVM J0: 0.73 .. 3406.48 = 3405.75s
[junit4] JVM J1: 0.95 .. 10255.73 = 10254.79s
[junit4] JVM J2: 0.95 .. 3397.47 = 3396.52s
[junit4] Execution time total: 2 hours 50 minutes 56 seconds
[junit4] Tests summary: 497 suites, 1967 tests, 4 suite-level errors, 1 error, 56 ignored (10 assumptions)
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:543: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:484: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/build.xml:61: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/extra-targets.xml:39: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/build.xml:229: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/solr/common-build.xml:511: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:1433: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.2/lucene/common-build.xml:991: There were test failures: 497 suites, 1967 tests, 4 suite-level errors, 1 error, 56 ignored (10 assumptions)
Total time: 284 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.2 #3
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 1583434417 bytes
Compression is 0.0%
Took 4 min 46 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure