You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/09/13 17:52:54 UTC

[JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 49 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/49/
Java: multiarch/jdk1.7.0 -d32 -server -XX:+UseConcMarkSweepGC

38 tests failed.
FAILED:  org.apache.solr.cloud.AsyncMigrateRouteKeyTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
Unable to create core [testcore]

Stack Trace:
org.apache.solr.common.SolrException: Unable to create core [testcore]
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:366D0E4B28D0C9F1]:0)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:741)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:701)
	at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:163)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper:conf1
	at org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:164)
	at org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:37)
	at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:71)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:725)
	... 41 more


FAILED:  org.apache.solr.cloud.CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:D082E48557C06F10]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:45D8F3171E0BB126]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:2586DCEF4C956FB1]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteShardTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistributedVersionInfoTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.OverseerStatusTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
	at org.apache.solr.cloud.RecoveryZkTest.distribTearDown(RecoveryZkTest.java:154)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
57 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=5093, name=qtp9060547-5093, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5128, name=zkCallback-705-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=5104, name=zkCallback-701-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5113, name=qtp6276707-5113, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=5054, name=qtp28558787-5054, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=5072, name=zkCallback-695-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=5106, name=Thread-2024, state=WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)    8) Thread[id=5126, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    9) Thread[id=5037, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=5083, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   11) Thread[id=5053, name=qtp28558787-5053-selector-ServerConnectorManager@c4e155/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=5091, name=qtp9060547-5091, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=5073, name=OverseerStateUpdate-94513373464297477-127.0.0.1:53763_o_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:170)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:144)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:193)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=5110, name=qtp6276707-5110-selector-ServerConnectorManager@614261/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=5079, name=coreContainerWorkExecutor-1550-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=5117, name=qtp6276707-5117, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=5088, name=qtp9060547-5088-selector-ServerConnectorManager@8e4ddb/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=5118, name=Scheduler-26157229, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=5114, name=qtp6276707-5114, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=5071, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   21) Thread[id=5115, name=qtp6276707-5115, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=5060, name=org.eclipse.jetty.server.session.HashSessionManager@1dd569bTimer, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=5081, name=zkCallback-695-thread-3, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=5082, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   25) Thread[id=5127, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   26) Thread[id=5055, name=qtp28558787-5055, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=5111, name=qtp6276707-5111-acceptor-0@14d2afe-ServerConnector@f770fd{SSL-http/1.1}{127.0.0.1:61264}, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=5112, name=qtp6276707-5112, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=5038, name=SessionTracker, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)   30) Thread[id=5077, name=Thread-2013, state=WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)   31) Thread[id=5102, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   32) Thread[id=5094, name=Scheduler-16259016, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=5059, name=Scheduler-28946995, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.c

[...truncated too long message...]

cher.process(LeaderElector.java:388)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:380)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:377)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:377)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:948)
   [junit4]   2> 	... 13 more
   [junit4]   2> 1187625 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@1549c2c{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1187626 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1b75598{/rq_,null,UNAVAILABLE}
   [junit4]   2> 1187629 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51832 51832
   [junit4]   2> 1187656 INFO  (Thread-5053) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51832 51832
   [junit4]   2> 1187657 WARN  (Thread-5053) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 	3	/solr/overseer/queue-work
   [junit4]   2> 	3	/solr/overseer/queue
   [junit4]   2> 	3	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionReloadTest -Dtests.method=testReloadedLeaderStateAfterZkSessionLoss -Dtests.seed=88E668E451AAC7C4 -Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=Asia/Jerusalem -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   26.9s J1 | CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1187660 INFO  (SUITE-CollectionReloadTest-seed#[88E668E451AAC7C4]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionReloadTest_88E668E451AAC7C4-001
   [junit4]   2> Sep 13, 2015 4:01:21 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=lt_LT, timezone=Asia/Jerusalem
   [junit4]   2> NOTE: SunOS 5.11 x86/Oracle Corporation 1.7.0_85 (32-bit)/cpus=3,threads=1,free=49072800,total=258502656
   [junit4]   2> NOTE: All tests run in this JVM: [SynonymTokenizerTest, TestFieldCollectionResource, TestUniqueKeyFieldResource, DirectSolrConnectionTest, CoreAdminCreateDiscoverTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, DebugComponentTest, UpdateRequestProcessorFactoryTest, SolrCoreCheckLockOnStartupTest, FacetPivotSmallTest, HdfsDirectoryFactoryTest, SpatialHeatmapFacetsTest, TestBlobHandler, SuggesterWFSTTest, QueryParsingTest, TestSimpleQParserPlugin, StressHdfsTest, TestSolrDeletionPolicy2, TestAuthenticationFramework, FileBasedSpellCheckerTest, BufferStoreTest, UniqFieldsUpdateProcessorFactoryTest, OverseerCollectionConfigSetProcessorTest, TestDistributedGrouping, CustomCollectionTest, TestTrie, RulesTest, ConcurrentDeleteAndCreateCollectionTest, NoCacheHeaderTest, ChangedSchemaMergeTest, CloudExitableDirectoryReaderTest, TestSchemaSimilarityResource, TestClusterStateMutator, VMParamsZkACLAndCredentialsProvidersTest, HighlighterMaxOffsetTest, TestComplexPhraseQParserPlugin, BinaryUpdateRequestHandlerTest, CloudMLTQParserTest, TestBlendedInfixSuggestions, ShowFileRequestHandlerTest, OverseerStatusTest, AutoCommitTest, TestFuzzyAnalyzedSuggestions, BigEndianAscendingWordSerializerTest, TestSolrCoreProperties, URLClassifyProcessorTest, NumericFieldsTest, TestIntervalFaceting, TestObjectReleaseTracker, DistributedDebugComponentTest, SparseHLLTest, SolrCmdDistributorTest, OutOfBoxZkACLAndCredentialsProvidersTest, SliceStateTest, CSVRequestHandlerTest, TestHashQParserPlugin, DeleteReplicaTest, TestCoreDiscovery, HdfsThreadLeakTest, DistributedSuggestComponentTest, TestHdfsUpdateLog, RAMDirectoryFactoryTest, TestManagedSchema, TestSolr4Spatial, TestPKIAuthenticationPlugin, TestFileDictionaryLookup, TestPHPSerializedResponseWriter, BJQParserTest, SuggesterTest, LoggingHandlerTest, HdfsNNFailoverTest, LukeRequestHandlerTest, TestSolrCloudWithKerberosAlt, TestCollapseQParserPlugin, OutputWriterTest, DistributedIntervalFacetingTest, BlockDirectoryTest, FullHLLTest, FieldMutatingUpdateProcessorTest, CoreAdminHandlerTest, TestBinaryField, OverriddenZkACLAndCredentialsProvidersTest, ChaosMonkeySafeLeaderTest, SpatialRPTFieldTypeTest, TestQueryUtils, FullSolrCloudDistribCmdsTest, PKIAuthenticationIntegrationTest, NotRequiredUniqueKeyTest, TestBM25SimilarityFactory, DistanceFunctionTest, TestQuerySenderListener, SolrPluginUtilsTest, TestReversedWildcardFilterFactory, TestCustomDocTransformer, TestFieldTypeResource, TestTrackingShardHandlerFactory, TestReplicationHandlerBackup, DistributedTermsComponentTest, HighlighterConfigTest, TestJoin, TestSolrCLIRunExample, DeleteShardTest, SolrCloudExampleTest, HLLSerializationTest, TestRebalanceLeaders, TestSortByMinMaxFunction, TestSolrIndexConfig, AnalyticsMergeStrategyTest, AtomicUpdatesTest, TestRandomFaceting, TestManagedSynonymFilterFactory, TestCollationField, TestCryptoKeys, TestSolrConfigHandler, DistributedQueueTest, TriLevelCompositeIdRoutingTest, BitVectorTest, DistributedVersionInfoTest, BadIndexSchemaTest, TestNonDefinedSimilarityFactory, TestSolrConfigHandlerCloud, TermVectorComponentDistributedTest, TestSolrQueryParser, DocValuesMissingTest, DefaultValueUpdateProcessorTest, TestTolerantSearch, TestSchemaManager, VersionInfoTest, UUIDFieldTest, CollectionReloadTest]
   [junit4] Completed [538/538] on J1 in 27.74s, 1 test, 1 error <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:729: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:524: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1452: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1006: There were test failures: 538 suites, 2127 tests, 2 suite-level errors, 36 errors, 1170 ignored (25 assumptions)

Total time: 44 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



RE: [JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 49 - Failure!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

 

I am very sorry for that. I reverted the change that lead to this. Problem was two things:

 

-          I did not notice the problem, because all tests affected by this were marked @Slow

-          The base class changes were correct to fix problems with RamUsageTester, BUT: After the test was running all static fields including SOLR_HOME were nulled. Unfortunately, the same base class was used as a static utility class by like 40 tests (the ones who failed). Those failed to build Zookeeper because SOLR_HOME and other stuff was missing.

 

To fix this we should move the “utility stuff” to a helper class that is not RamUsageTested.

 

I will also discuss with Dawid how to make RamUsageTester not complain if it cannot access classes that are part of Java 9’s module system and not accessible for ram usage testing.

 

Uwe

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

 <http://www.thetaphi.de/> http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Uwe Schindler [mailto:uwe@thetaphi.de] 
Sent: Sunday, September 13, 2015 6:19 PM
To: dev@lucene.apache.org
Subject: Re: [JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 49 - Failure!

 

Hi,

Could be related to my changes. The test passed fully on Windows. Could be the changes to some BaseDistributed+ stuff.

I am away from computer so can only look into it later!

Sorry,
Uwe

Am 13. September 2015 17:52:54 MESZ, schrieb Policeman Jenkins Server <je...@thetaphi.de>:

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/49/
Java: multiarch/jdk1.7.0 -d32 -server -XX:+UseConcMarkSweepGC

38 tests failed.
FAILED:  org.apache.solr.cloud.AsyncMigrateRouteKeyTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
No registered
leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)


at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED: 
org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
Unable to create core [testcore]

Stack Trace:
org.apache.solr.common.SolrException: Unable to create core [testcore]
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:366D0E4B28D0C9F1]:0)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:741)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:701)
 at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:163)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper:conf1
 at org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:164)
 at
org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:37)
 at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:71)
 at org.apache.solr.core.CoreContainer.create(CoreContainer.java:725)
 ... 41 more


FAILED:  org.apache.solr.cloud.CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls

Error
Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:D082E48557C06F10]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)


at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED: 
org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:45D8F3171E0BB126]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:2586DCEF4C956FB1]:0)
 at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteShardTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistributedVersionInfoTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack
Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
No
registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)


at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.OverseerStatusTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED: 
org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
 at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
 at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:


Stack Trace:
java.lang.NullPointerException
 at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
 at org.apache.solr.cloud.RecoveryZkTest.distribTearDown(RecoveryZkTest.java:154)
 at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
 at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
 at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
 at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
 at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
 at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
 at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
 at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)


at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
 at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
 at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
 at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
57 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=5093, name=qtp9060547-5093, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5128, name=zkCallback-705-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=5104, name=zkCallback-701-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5113, name=qtp6276707-5113, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    5)
Thread[id=5054, name=qtp28558787-5054, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=5072, name=zkCallback-695-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)    
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=5106, name=Thread-2024, state=WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)    8) Thread[id=5126, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141),
state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)    9) Thread[id=5037, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=5083, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   11) Thread[id=5053, name=qtp28558787-5053-selector-ServerConnectorManager@c4e155/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=5091, name=qtp9060547-5091, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=5073, name=OverseerStateUpdate-94513373464297477-127.0.0.1:53763_o_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:170)         at
org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:144)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:193)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=5110, name=qtp6276707-5110-selector-ServerConnectorManager@614261/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=5079, name=coreContainerWorkExecutor-1550-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=5117, name=qtp6276707-5117, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=5088, name=qtp9060547-5088-selector-ServerConnectorManager@8e4ddb/0, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)         at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)         at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=5118, name=Scheduler-26157229, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=5114, name=qtp6276707-5114, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=5071, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   21) Thread[id=5115, name=qtp6276707-5115, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=5060,
name=org.eclipse.jetty.server.session.HashSessionManager@1dd569bTimer, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=5081, name=zkCallback-695-thread-3,
state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=5082, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   25) Thread[id=5127, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   26) Thread[id=5055, name=qtp28558787-5055, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=5111, name=qtp6276707-5111-acceptor-0@14d2afe-ServerConnector@f770fd{SSL-http/1.1}{127.0.0.1:61264}, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)         at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=5112, name=qtp6276707-5112,
state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=5038, name=SessionTracker, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)   30) Thread[id=5077, name=Thread-2013,
state=WAITING, group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)   31) Thread[id=5102, name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141), state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   32) Thread[id=5094, name=Scheduler-16259016, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=5059, name=Scheduler-28946995, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.c

[...truncated too long message...]

cher.process(LeaderElector.java:388)
   [junit4]   2>  at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
   [junit4]   2>  at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2>  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2>  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2>  at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2>  at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2>  at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2>  at
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2>  at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:380)
   [junit4]   2>  at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:377)
   [junit4]   2>  at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2>  at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:377)
   [junit4]   2>  at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:948)
   [junit4]   2>  ... 13 more
   [junit4]   2> 1187625 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@1549c2c{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1187626 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ]
o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1b75598{/rq_,null,UNAVAILABLE}
   [junit4]   2> 1187629 INFO  (TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51832 51832
   [junit4]   2> 1187656 INFO  (Thread-5053) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51832 51832
   [junit4]   2> 1187657 WARN  (Thread-5053) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>  6 /solr/aliases.json
   [junit4]   2>  5 /solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>  6 /solr/clusterstate.json
   [junit4]   2>  4
/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>  6 /solr/live_nodes
   [junit4]   2>  6 /solr/collections
   [junit4]   2>  3 /solr/overseer/queue-work
   [junit4]   2>  3 /solr/overseer/queue
   [junit4]   2>  3 /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionReloadTest -Dtests.method=testReloadedLeaderStateAfterZkSessionLoss -Dtests.seed=88E668E451AAC7C4 -Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=Asia/Jerusalem -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   26.9s J1 | CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms
, collection: collection1 slice: shard1
   [junit4]    >  at __randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
   [junit4]    >  at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
   [junit4]    >  at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
   [junit4]    >  at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
   [junit4]    >  at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
   [junit4]    >  at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
   [junit4]    >  at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1187660 INFO 
(SUITE-CollectionReloadTest-seed#[88E668E451AAC7C4]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionReloadTest_88E668E451AAC7C4-001
   [junit4]   2> Sep 13, 2015 4:01:21 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=lt_LT, timezone=Asia/Jerusalem
   [junit4]   2> NOTE: SunOS 5.11 x86/Oracle Corporation 1.7.0_85 (32-bit)/cpus=3,threads=1,free=49072800,total=258502656
   [junit4]   2> NOTE: All tests run in this JVM: [SynonymTokenizerTest, TestFieldCollectionResource, TestUniqueKeyFieldResource,
DirectSolrConnectionTest, CoreAdminCreateDiscoverTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, DebugComponentTest, UpdateRequestProcessorFactoryTest, SolrCoreCheckLockOnStartupTest, FacetPivotSmallTest, HdfsDirectoryFactoryTest, SpatialHeatmapFacetsTest, TestBlobHandler, SuggesterWFSTTest, QueryParsingTest, TestSimpleQParserPlugin, StressHdfsTest, TestSolrDeletionPolicy2, TestAuthenticationFramework, FileBasedSpellCheckerTest, BufferStoreTest, UniqFieldsUpdateProcessorFactoryTest, OverseerCollectionConfigSetProcessorTest, TestDistributedGrouping, CustomCollectionTest, TestTrie, RulesTest, ConcurrentDeleteAndCreateCollectionTest, NoCacheHeaderTest, ChangedSchemaMergeTest, CloudExitableDirectoryReaderTest, TestSchemaSimilarityResource, TestClusterStateMutator, VMParamsZkACLAndCredentialsProvidersTest, HighlighterMaxOffsetTest, TestComplexPhraseQParserPlugin, BinaryUpdateRequestHandlerTest, CloudMLTQParserTest, TestBlendedInfixSuggestions, ShowFileRequestHandlerTest,
OverseerStatusTest, AutoCommitTest, TestFuzzyAnalyzedSuggestions, BigEndianAscendingWordSerializerTest, TestSolrCoreProperties, URLClassifyProcessorTest, NumericFieldsTest, TestIntervalFaceting, TestObjectReleaseTracker, DistributedDebugComponentTest, SparseHLLTest, SolrCmdDistributorTest, OutOfBoxZkACLAndCredentialsProvidersTest, SliceStateTest, CSVRequestHandlerTest, TestHashQParserPlugin, DeleteReplicaTest, TestCoreDiscovery, HdfsThreadLeakTest, DistributedSuggestComponentTest, TestHdfsUpdateLog, RAMDirectoryFactoryTest, TestManagedSchema, TestSolr4Spatial, TestPKIAuthenticationPlugin, TestFileDictionaryLookup, TestPHPSerializedResponseWriter, BJQParserTest, SuggesterTest, LoggingHandlerTest, HdfsNNFailoverTest, LukeRequestHandlerTest, TestSolrCloudWithKerberosAlt, TestCollapseQParserPlugin, OutputWriterTest, DistributedIntervalFacetingTest, BlockDirectoryTest, FullHLLTest, FieldMutatingUpdateProcessorTest, CoreAdminHandlerTest, TestBinaryField,
OverriddenZkACLAndCredentialsProvidersTest, ChaosMonkeySafeLeaderTest, SpatialRPTFieldTypeTest, TestQueryUtils, FullSolrCloudDistribCmdsTest, PKIAuthenticationIntegrationTest, NotRequiredUniqueKeyTest, TestBM25SimilarityFactory, DistanceFunctionTest, TestQuerySenderListener, SolrPluginUtilsTest, TestReversedWildcardFilterFactory, TestCustomDocTransformer, TestFieldTypeResource, TestTrackingShardHandlerFactory, TestReplicationHandlerBackup, DistributedTermsComponentTest, HighlighterConfigTest, TestJoin, TestSolrCLIRunExample, DeleteShardTest, SolrCloudExampleTest, HLLSerializationTest, TestRebalanceLeaders, TestSortByMinMaxFunction, TestSolrIndexConfig, AnalyticsMergeStrategyTest, AtomicUpdatesTest, TestRandomFaceting, TestManagedSynonymFilterFactory, TestCollationField, TestCryptoKeys, TestSolrConfigHandler, DistributedQueueTest, TriLevelCompositeIdRoutingTest, BitVectorTest, DistributedVersionInfoTest, BadIndexSchemaTest, TestNonDefinedSimilarityFactory, TestSolrConfigHandlerCloud,
TermVectorComponentDistributedTest, TestSolrQueryParser, DocValuesMissingTest, DefaultValueUpdateProcessorTest, TestTolerantSearch, TestSchemaManager, VersionInfoTest, UUIDFieldTest, CollectionReloadTest]
   [junit4] Completed [538/538] on J1 in 27.74s, 1 test, 1 error <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:729: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:524: The following error occurred
while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1452: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1006: There were test failures: 538 suites, 2127 tests, 2 suite-level errors, 36 errors, 1170 ignored (25 assumptions)

Total time: 44 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


 

  _____  


To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de


Re: [JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 49 - Failure!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

Could be related to my changes. The test passed fully on Windows. Could be the changes to some BaseDistributed+ stuff.

I am away from computer so can only look into it later!

Sorry,
Uwe

Am 13. September 2015 17:52:54 MESZ, schrieb Policeman Jenkins Server <je...@thetaphi.de>:
>Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/49/
>Java: multiarch/jdk1.7.0 -d32 -server -XX:+UseConcMarkSweepGC
>
>38 tests failed.
>FAILED:  org.apache.solr.cloud.AsyncMigrateRouteKeyTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED: 
>org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration
>
>Error Message:
>Unable to create core [testcore]
>
>Stack Trace:
>org.apache.solr.common.SolrException: Unable to create core [testcore]
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:366D0E4B28D0C9F1]:0)
>	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:741)
>	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:701)
>	at
>org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:163)
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>	at
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>	at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:606)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified
>config does not exist in ZooKeeper:conf1
>	at
>org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:164)
>	at
>org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:37)
>	at
>org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:71)
>	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:725)
>	... 41 more
>
>
>FAILED: 
>org.apache.solr.cloud.CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED: 
>org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:D082E48557C06F10]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.CustomCollectionTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED: 
>org.apache.solr.cloud.DeleteInactiveReplicaTest.deleteInactiveReplicaTest
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:45D8F3171E0BB126]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:2586DCEF4C956FB1]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.DeleteShardTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.DistributedVersionInfoTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.OverseerStatusTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test
>
>Error Message:
>No registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>
>Stack Trace:
>org.apache.solr.common.SolrException: No registered leader was found
>after waiting for 10000ms , collection: collection1 slice: shard1
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  org.apache.solr.cloud.RecoveryZkTest.test
>
>Error Message:
>
>
>Stack Trace:
>java.lang.NullPointerException
>	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:B2573EFF56AA3C]:0)
>	at
>org.apache.solr.cloud.RecoveryZkTest.distribTearDown(RecoveryZkTest.java:154)
>	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>	at
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>	at
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>	at
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>	at
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>	at
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>	at
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>	at
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>	at java.lang.Thread.run(Thread.java:745)
>
>
>FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
>
>Error Message:
>57 threads leaked from SUITE scope at
>org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=5093,
>name=qtp9060547-5093, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)    2) Thread[id=5128,
>name=zkCallback-705-thread-1, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>at
>java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>at
>java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)  
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)    3) Thread[id=5104,
>name=zkCallback-701-thread-1, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>at
>java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>at
>java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)  
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)    4) Thread[id=5113,
>name=qtp6276707-5113, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)    5) Thread[id=5054,
>name=qtp28558787-5054, state=TIMED_WAITING, group=TGRP-RecoveryZkTest] 
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)    6) Thread[id=5072,
>name=zkCallback-695-thread-1, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>at
>java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>at
>java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)  
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)    7) Thread[id=5106,
>name=Thread-2024, state=WAITING, group=TGRP-RecoveryZkTest]         at
>java.lang.Object.wait(Native Method)         at
>java.lang.Object.wait(Object.java:503)         at
>org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)    8)
>Thread[id=5126,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141),
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>9) Thread[id=5037, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0,
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
>at java.lang.Thread.run(Thread.java:745)   10) Thread[id=5083,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread,
>state=WAITING, group=TGRP-RecoveryZkTest]         at
>sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)      
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>at
>java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>11) Thread[id=5053,
>name=qtp28558787-5053-selector-ServerConnectorManager@c4e155/0,
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
>at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
>at
>org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>at java.lang.Thread.run(Thread.java:745)   12) Thread[id=5091,
>name=qtp9060547-5091, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   13) Thread[id=5073,
>name=OverseerStateUpdate-94513373464297477-127.0.0.1:53763_o_-n_0000000000,
>state=TIMED_WAITING, group=Overseer state updater.]         at
>sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:170) 
>at
>org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:144) 
>at
>org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:193)
>at java.lang.Thread.run(Thread.java:745)   14) Thread[id=5110,
>name=qtp6276707-5110-selector-ServerConnectorManager@614261/0,
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
>at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
>at
>org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>at java.lang.Thread.run(Thread.java:745)   15) Thread[id=5079,
>name=coreContainerWorkExecutor-1550-thread-1, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>at
>java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>at
>java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)  
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)   16) Thread[id=5117,
>name=qtp6276707-5117, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   17) Thread[id=5088,
>name=qtp9060547-5088-selector-ServerConnectorManager@8e4ddb/0,
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)         at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
>at
>org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
>at
>org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>at java.lang.Thread.run(Thread.java:745)   18) Thread[id=5118,
>name=Scheduler-26157229, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)   19) Thread[id=5114,
>name=qtp6276707-5114, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   20) Thread[id=5071,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread,
>state=WAITING, group=TGRP-RecoveryZkTest]         at
>sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)      
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>at
>java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>21) Thread[id=5115, name=qtp6276707-5115, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   22) Thread[id=5060,
>name=org.eclipse.jetty.server.session.HashSessionManager@1dd569bTimer,
>state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at
>sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)   23) Thread[id=5081,
>name=zkCallback-695-thread-3, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>at
>java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
>at
>java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)  
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)   24) Thread[id=5082,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141),
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>25) Thread[id=5127,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-EventThread,
>state=WAITING, group=TGRP-RecoveryZkTest]         at
>sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)      
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>at
>java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>26) Thread[id=5055, name=qtp28558787-5055, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   27) Thread[id=5111,
>name=qtp6276707-5111-acceptor-0@14d2afe-ServerConnector@f770fd{SSL-http/1.1}{127.0.0.1:61264},
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at
>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
>at
>org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
>at
>org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>at java.lang.Thread.run(Thread.java:745)   28) Thread[id=5112,
>name=qtp6276707-5112, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]  
>at sun.misc.Unsafe.park(Native Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
>at
>org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
>at java.lang.Thread.run(Thread.java:745)   29) Thread[id=5038,
>name=SessionTracker, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]   
>at java.lang.Object.wait(Native Method)         at
>org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
>30) Thread[id=5077, name=Thread-2013, state=WAITING,
>group=TGRP-RecoveryZkTest]         at java.lang.Object.wait(Native
>Method)         at java.lang.Object.wait(Object.java:503)         at
>org.apache.solr.core.CloserThread.run(CoreContainer.java:1153)   31)
>Thread[id=5102,
>name=TEST-RecoveryZkTest.test-seed#[88E668E451AAC7C4]-SendThread(127.0.0.1:61141),
>state=RUNNABLE, group=TGRP-RecoveryZkTest]         at
>sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)         at
>sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)      
>at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:84)
>at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)       
>at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at
>org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>32) Thread[id=5094, name=Scheduler-16259016, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>at
>java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>at java.lang.Thread.run(Thread.java:745)   33) Thread[id=5059,
>name=Scheduler-28946995, state=TIMED_WAITING,
>group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native
>Method)         at
>java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
>at
>java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>at
>java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>        at java.util.c
>
>[...truncated too long message...]
>
>cher.process(LeaderElector.java:388)
>[junit4]   2> 	at
>org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
>[junit4]   2> 	at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>[junit4]   2> 	at
>java.util.concurrent.FutureTask.run(FutureTask.java:262)
>[junit4]   2> 	at
>org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
>[junit4]   2> 	at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>[junit4]   2> 	at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
>[junit4]   2> Caused by:
>org.apache.zookeeper.KeeperException$SessionExpiredException:
>KeeperErrorCode = Session expired for /overseer
>[junit4]   2> 	at
>org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>[junit4]   2> 	at
>org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>[junit4]   2> 	at
>org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>[junit4]   2> 	at
>org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:380)
>[junit4]   2> 	at
>org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:377)
>[junit4]   2> 	at
>org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
>[junit4]   2> 	at
>org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:377)
>[junit4]   2> 	at
>org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:948)
>   [junit4]   2> 	... 13 more
>[junit4]   2> 1187625 INFO 
>(TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4])
>[    ] o.e.j.s.ServerConnector Stopped
>ServerConnector@1549c2c{HTTP/1.1}{127.0.0.1:0}
>[junit4]   2> 1187626 INFO 
>(TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4])
>[    ] o.e.j.s.h.ContextHandler Stopped
>o.e.j.s.ServletContextHandler@1b75598{/rq_,null,UNAVAILABLE}
>[junit4]   2> 1187629 INFO 
>(TEST-CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss-seed#[88E668E451AAC7C4])
>[    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51832 51832
>[junit4]   2> 1187656 INFO  (Thread-5053) [    ] o.a.s.c.ZkTestServer
>connecting to 127.0.0.1:51832 51832
>[junit4]   2> 1187657 WARN  (Thread-5053) [    ] o.a.s.c.ZkTestServer
>Watch limit violations: 
>   [junit4]   2> Maximum concurrent create/delete watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	6	/solr/aliases.json
>   [junit4]   2> 	5	/solr/security.json
>   [junit4]   2> 
>   [junit4]   2> Maximum concurrent data watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	6	/solr/clusterstate.json
>   [junit4]   2> 	4	/solr/collections/collection1/state.json
>   [junit4]   2> 
>   [junit4]   2> Maximum concurrent children watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	6	/solr/live_nodes
>   [junit4]   2> 	6	/solr/collections
>   [junit4]   2> 	3	/solr/overseer/queue-work
>   [junit4]   2> 	3	/solr/overseer/queue
>   [junit4]   2> 	3	/solr/overseer/collection-queue-work
>   [junit4]   2> 
>[junit4]   2> NOTE: reproduce with: ant test 
>-Dtestcase=CollectionReloadTest
>-Dtests.method=testReloadedLeaderStateAfterZkSessionLoss
>-Dtests.seed=88E668E451AAC7C4 -Dtests.slow=true -Dtests.locale=lt_LT
>-Dtests.timezone=Asia/Jerusalem -Dtests.asserts=true
>-Dtests.file.encoding=US-ASCII
>[junit4] ERROR   26.9s J1 |
>CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss <<<
>[junit4]    > Throwable #1: org.apache.solr.common.SolrException: No
>registered leader was found after waiting for 10000ms , collection:
>collection1 slice: shard1
>[junit4]    > 	at
>__randomizedtesting.SeedInfo.seed([88E668E451AAC7C4:73C8D3B9F747B491]:0)
>[junit4]    > 	at
>org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
>[junit4]    > 	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
>[junit4]    > 	at
>org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
>[junit4]    > 	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
>[junit4]    > 	at
>org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
>   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
>[junit4]   2> 1187660 INFO 
>(SUITE-CollectionReloadTest-seed#[88E668E451AAC7C4]-worker) [    ]
>o.a.s.SolrTestCaseJ4 ###deleteCore
>[junit4]   2> NOTE: leaving temporary files on disk at:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionReloadTest_88E668E451AAC7C4-001
>[junit4]   2> Sep 13, 2015 4:01:21 PM
>com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>[junit4]   2> WARNING: Will linger awaiting termination of 1 leaked
>thread(s).
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {},
>docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy):
>{}, locale=lt_LT, timezone=Asia/Jerusalem
>[junit4]   2> NOTE: SunOS 5.11 x86/Oracle Corporation 1.7.0_85
>(32-bit)/cpus=3,threads=1,free=49072800,total=258502656
>[junit4]   2> NOTE: All tests run in this JVM: [SynonymTokenizerTest,
>TestFieldCollectionResource, TestUniqueKeyFieldResource,
>DirectSolrConnectionTest, CoreAdminCreateDiscoverTest,
>TermVectorComponentTest, FieldAnalysisRequestHandlerTest,
>DebugComponentTest, UpdateRequestProcessorFactoryTest,
>SolrCoreCheckLockOnStartupTest, FacetPivotSmallTest,
>HdfsDirectoryFactoryTest, SpatialHeatmapFacetsTest, TestBlobHandler,
>SuggesterWFSTTest, QueryParsingTest, TestSimpleQParserPlugin,
>StressHdfsTest, TestSolrDeletionPolicy2, TestAuthenticationFramework,
>FileBasedSpellCheckerTest, BufferStoreTest,
>UniqFieldsUpdateProcessorFactoryTest,
>OverseerCollectionConfigSetProcessorTest, TestDistributedGrouping,
>CustomCollectionTest, TestTrie, RulesTest,
>ConcurrentDeleteAndCreateCollectionTest, NoCacheHeaderTest,
>ChangedSchemaMergeTest, CloudExitableDirectoryReaderTest,
>TestSchemaSimilarityResource, TestClusterStateMutator,
>VMParamsZkACLAndCredentialsProvidersTest, HighlighterMaxOffsetTest,
>TestComplexPhraseQParserPlugin, BinaryUpdateRequestHandlerTest,
>CloudMLTQParserTest, TestBlendedInfixSuggestions,
>ShowFileRequestHandlerTest, OverseerStatusTest, AutoCommitTest,
>TestFuzzyAnalyzedSuggestions, BigEndianAscendingWordSerializerTest,
>TestSolrCoreProperties, URLClassifyProcessorTest, NumericFieldsTest,
>TestIntervalFaceting, TestObjectReleaseTracker,
>DistributedDebugComponentTest, SparseHLLTest, SolrCmdDistributorTest,
>OutOfBoxZkACLAndCredentialsProvidersTest, SliceStateTest,
>CSVRequestHandlerTest, TestHashQParserPlugin, DeleteReplicaTest,
>TestCoreDiscovery, HdfsThreadLeakTest, DistributedSuggestComponentTest,
>TestHdfsUpdateLog, RAMDirectoryFactoryTest, TestManagedSchema,
>TestSolr4Spatial, TestPKIAuthenticationPlugin,
>TestFileDictionaryLookup, TestPHPSerializedResponseWriter,
>BJQParserTest, SuggesterTest, LoggingHandlerTest, HdfsNNFailoverTest,
>LukeRequestHandlerTest, TestSolrCloudWithKerberosAlt,
>TestCollapseQParserPlugin, OutputWriterTest,
>DistributedIntervalFacetingTest, BlockDirectoryTest, FullHLLTest,
>FieldMutatingUpdateProcessorTest, CoreAdminHandlerTest,
>TestBinaryField, OverriddenZkACLAndCredentialsProvidersTest,
>ChaosMonkeySafeLeaderTest, SpatialRPTFieldTypeTest, TestQueryUtils,
>FullSolrCloudDistribCmdsTest, PKIAuthenticationIntegrationTest,
>NotRequiredUniqueKeyTest, TestBM25SimilarityFactory,
>DistanceFunctionTest, TestQuerySenderListener, SolrPluginUtilsTest,
>TestReversedWildcardFilterFactory, TestCustomDocTransformer,
>TestFieldTypeResource, TestTrackingShardHandlerFactory,
>TestReplicationHandlerBackup, DistributedTermsComponentTest,
>HighlighterConfigTest, TestJoin, TestSolrCLIRunExample,
>DeleteShardTest, SolrCloudExampleTest, HLLSerializationTest,
>TestRebalanceLeaders, TestSortByMinMaxFunction, TestSolrIndexConfig,
>AnalyticsMergeStrategyTest, AtomicUpdatesTest, TestRandomFaceting,
>TestManagedSynonymFilterFactory, TestCollationField, TestCryptoKeys,
>TestSolrConfigHandler, DistributedQueueTest,
>TriLevelCompositeIdRoutingTest, BitVectorTest,
>DistributedVersionInfoTest, BadIndexSchemaTest,
>TestNonDefinedSimilarityFactory, TestSolrConfigHandlerCloud,
>TermVectorComponentDistributedTest, TestSolrQueryParser,
>DocValuesMissingTest, DefaultValueUpdateProcessorTest,
>TestTolerantSearch, TestSchemaManager, VersionInfoTest, UUIDFieldTest,
>CollectionReloadTest]
>[junit4] Completed [538/538] on J1 in 27.74s, 1 test, 1 error <<<
>FAILURES!
>
>[...truncated 19 lines...]
>BUILD FAILED
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:729:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:524:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1452:
>The following error occurred while executing this line:
>/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1006:
>There were test failures: 538 suites, 2127 tests, 2 suite-level errors,
>36 errors, 1170 ignored (25 assumptions)
>
>Total time: 44 minutes 12 seconds
>Build step 'Invoke Ant' marked build as failure
>Archiving artifacts
>[WARNINGS] Skipping publisher since build result is FAILURE
>Recording test results
>Email was triggered for: Failure - Any
>Sending email for trigger: Failure - Any
>
>
>
>
>------------------------------------------------------------------------
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>For additional commands, e-mail: dev-help@lucene.apache.org

--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de

[JENKINS] Lucene-Solr-5.x-Solaris (64bit/jdk1.8.0) - Build # 51 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/51/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestConfigSetsAPIExclusivity.testAPIExclusivity

Error Message:
Unexpected exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50507/solr: delete the configset time out:300s expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: Unexpected exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50507/solr: delete the configset time out:300s expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([72ED523DC3CC4F54:F984BC69A68F61D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestConfigSetsAPIExclusivity.testAPIExclusivity(TestConfigSetsAPIExclusivity.java:95)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10757 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestConfigSetsAPIExclusivity
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/init-core-data-001
   [junit4]   2> 2077627 INFO  (SUITE-TestConfigSetsAPIExclusivity-seed#[72ED523DC3CC4F54]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2077632 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testAPIExclusivity
   [junit4]   2> 2077634 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2077634 INFO  (Thread-7134) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2077634 INFO  (Thread-7134) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2077739 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.ZkTestServer start zk server on port:50396
   [junit4]   2> 2077739 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2077740 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077746 INFO  (zkCallback-1889-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@783973a5 name:ZooKeeperConnection Watcher:127.0.0.1:50396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077746 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077747 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2077747 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 2077755 INFO  (jetty-launcher-1888-thread-1) [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 2077761 INFO  (jetty-launcher-1888-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@173cdfde{/solr,null,AVAILABLE}
   [junit4]   2> 2077762 INFO  (jetty-launcher-1888-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@64292560{HTTP/1.1}{127.0.0.1:50507}
   [junit4]   2> 2077762 INFO  (jetty-launcher-1888-thread-1) [    ] o.e.j.s.Server Started @2111496ms
   [junit4]   2> 2077763 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50507}
   [junit4]   2> 2077763 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@659e0bfd
   [junit4]   2> 2077763 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/'
   [junit4]   2> 2077782 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2077783 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077785 INFO  (zkCallback-1890-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@606487b4 name:ZooKeeperConnection Watcher:127.0.0.1:50396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077786 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077786 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2077787 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2077795 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.CoresLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/.
   [junit4]   2> 2077796 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 924023624
   [junit4]   2> 2077796 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/]
   [junit4]   2> 2077796 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.CoreContainer loading shared library: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/lib
   [junit4]   2> 2077796 WARN  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to classloader: lib (resolved as: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/lib).
   [junit4]   2> 2077808 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2077809 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2077810 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2077810 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2077811 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50396/solr
   [junit4]   2> 2077811 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2077811 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2077811 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077814 INFO  (zkCallback-1892-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7605748a name:ZooKeeperConnection Watcher:127.0.0.1:50396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077814 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077815 INFO  (jetty-launcher-1888-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2077816 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x14fca16e28c0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2077817 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077819 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@871af8d name:ZooKeeperConnection Watcher:127.0.0.1:50396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077819 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077821 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2077825 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2077830 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2077834 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2077838 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2077841 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2077843 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 2077846 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2077848 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2077850 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2077852 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50507_solr
   [junit4]   2> 2077853 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50507_solr
   [junit4]   2> 2077856 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2077858 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2077861 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2077862 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/94516316836986883-127.0.0.1:50507_solr-n_0000000000
   [junit4]   2> 2077863 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50507_solr
   [junit4]   2> 2077863 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2077865 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.Overseer Overseer (id=94516316836986883-127.0.0.1:50507_solr-n_0000000000) starting
   [junit4]   2> 2077868 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2077879 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2077880 INFO  (OverseerStateUpdate-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2077880 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2077884 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 2077899 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2077899 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2077901 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.CoresLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001/tempDir-001/.
   [junit4]   2> 2077902 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.c.CoresLocator Found 0 core definitions
   [junit4]   2> 2077904 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 2077904 INFO  (jetty-launcher-1888-thread-1) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2077905 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2077906 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077908 INFO  (zkCallback-1894-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@757aa44f name:ZooKeeperConnection Watcher:127.0.0.1:50396 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077909 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077909 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2077914 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2077915 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2077917 INFO  (zkCallback-1896-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@576c0a0f name:ZooKeeperConnection Watcher:127.0.0.1:50396/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2077917 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2077918 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2077919 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/grandBaseConfigSet1/schema.xml
   [junit4]   2> 2077923 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/grandBaseConfigSet1/solrconfig.xml
   [junit4]   2> 2077947 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2077950 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2077950 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2077950 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2077955 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2077956 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000004 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2077960 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2077960 INFO  (OverseerThreadFactory-6538-thread-2-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2077962 ERROR (OverseerThreadFactory-6538-thread-2-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2077963 INFO  (OverseerThreadFactory-6538-thread-2-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000004 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2077963 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2077964 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2077967 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000004 state SyncConnected
   [junit4]   2> 2077970 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=22 
   [junit4]   2> 2077970 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2077972 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2077973 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2077975 INFO  (OverseerThreadFactory-6538-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{}
   [junit4]   2> 2077977 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2077980 INFO  (OverseerThreadFactory-6538-thread-3-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2077980 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000000 state SyncConnected
   [junit4]   2> 2077981 INFO  (OverseerThreadFactory-6538-thread-3-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2077982 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=35 
   [junit4]   2> 2077984 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2077987 INFO  (OverseerThreadFactory-6538-thread-3-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000002 complete, response:{}
   [junit4]   2> 2077988 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000005 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2077990 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2077990 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000002 state SyncConnected
   [junit4]   2> 2077991 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: create failed:org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.createConfigSet(OverseerConfigSetMessageHandler.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2077992 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000005 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1,exception={msg=Base ConfigSet does not exist: baseConfigSet1,rspCode=400}}
   [junit4]   2> 2077992 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=45 
   [junit4]   2> 2077995 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2077996 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000005 state SyncConnected
   [junit4]   2> 2077999 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2} status=400 QTime=52 
   [junit4]   2> 2078000 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000008 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078001 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000010 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078001 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078002 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078002 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078004 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078005 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000008 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078006 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078006 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078007 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000008 state SyncConnected
   [junit4]   2> 2078010 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=37 
   [junit4]   2> 2078011 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078012 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078012 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078017 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000010 complete, response:{}
   [junit4]   2> 2078018 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000010 state SyncConnected
   [junit4]   2> 2078020 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=36 
   [junit4]   2> 2078021 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000012 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078022 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078022 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078024 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2078028 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000012 complete, response:{}
   [junit4]   2> 2078031 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000012 state SyncConnected
   [junit4]   2> 2078034 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=39 
   [junit4]   2> 2078036 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078037 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000014 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078037 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078039 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: create failed:org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.createConfigSet(OverseerConfigSetMessageHandler.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078040 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000014 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1,exception={msg=Base ConfigSet does not exist: baseConfigSet1,rspCode=400}}
   [junit4]   2> 2078040 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000016 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078040 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000018 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078040 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078041 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078042 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078043 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000014 state SyncConnected
   [junit4]   2> 2078045 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078045 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000016 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078046 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078047 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2} status=400 QTime=45 
   [junit4]   2> 2078049 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078049 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000016 state SyncConnected
   [junit4]   2> 2078051 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078053 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078053 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=41 
   [junit4]   2> 2078055 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078057 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000018 complete, response:{}
   [junit4]   2> 2078059 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000018 state SyncConnected
   [junit4]   2> 2078061 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=39 
   [junit4]   2> 2078063 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078065 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000020 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078065 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000024 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078065 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078066 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078067 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078068 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000024 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078068 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2078071 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000024 state SyncConnected
   [junit4]   2> 2078073 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000020 complete, response:{}
   [junit4]   2> 2078074 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=18 
   [junit4]   2> 2078074 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000022 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078075 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078076 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078077 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: create failed:org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.createConfigSet(OverseerConfigSetMessageHandler.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078077 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000020 state SyncConnected
   [junit4]   2> 2078079 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000022 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1,exception={msg=Base ConfigSet does not exist: baseConfigSet1,rspCode=400}}
   [junit4]   2> 2078080 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=43 
   [junit4]   2> 2078082 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078085 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000022 state SyncConnected
   [junit4]   2> 2078088 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2} status=400 QTime=39 
   [junit4]   2> 2078091 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078095 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000026 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078095 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000028 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078095 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078096 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078097 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078098 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000028 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078099 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078100 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078100 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000028 state SyncConnected
   [junit4]   2> 2078104 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=28 
   [junit4]   2> 2078106 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078107 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078108 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078112 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000026 complete, response:{}
   [junit4]   2> 2078112 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000030 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078113 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078118 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2078119 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000026 state SyncConnected
   [junit4]   2> 2078121 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=58 
   [junit4]   2> 2078124 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078127 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000030 complete, response:{}
   [junit4]   2> 2078132 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000030 state SyncConnected
   [junit4]   2> 2078133 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=51 
   [junit4]   2> 2078135 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078139 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000032 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078139 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078142 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: create failed:org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.createConfigSet(OverseerConfigSetMessageHandler.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078143 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000032 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1,exception={msg=Base ConfigSet does not exist: baseConfigSet1,rspCode=400}}
   [junit4]   2> 2078144 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000034 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078144 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000036 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078145 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078145 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078147 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078150 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000032 state SyncConnected
   [junit4]   2> 2078153 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000034 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078155 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2} status=400 QTime=64 
   [junit4]   2> 2078156 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078157 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078158 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000034 state SyncConnected
   [junit4]   2> 2078160 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078162 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=56 
   [junit4]   2> 2078163 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078165 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078165 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078172 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000036 complete, response:{}
   [junit4]   2> 2078177 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000036 state SyncConnected
   [junit4]   2> 2078179 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=55 
   [junit4]   2> 2078182 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078184 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000038 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078185 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000042 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078185 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078186 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078186 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2078187 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078187 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000042 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078190 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000042 state SyncConnected
   [junit4]   2> 2078193 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=28 
   [junit4]   2> 2078193 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000038 complete, response:{}
   [junit4]   2> 2078194 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000040 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078195 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "baseConfigSet":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078195 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078198 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000038 state SyncConnected
   [junit4]   2> 2078198 ERROR (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: create failed:org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.createConfigSet(OverseerConfigSetMessageHandler.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078199 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000040 complete, response:{Operation create caused exception:=org.apache.solr.common.SolrException: Base ConfigSet does not exist: baseConfigSet1,exception={msg=Base ConfigSet does not exist: baseConfigSet1,rspCode=400}}
   [junit4]   2> 2078200 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=65 
   [junit4]   2> 2078203 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=baseConfigSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078203 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000040 state SyncConnected
   [junit4]   2> 2078206 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2} status=400 QTime=46 
   [junit4]   2> 2078209 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000044 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078209 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000046 message:{
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078209 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "baseConfigSet":"grandBaseConfigSet1",
   [junit4]   2>   "operation":"configsets:create"}
   [junit4]   2> 2078209 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"configSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078210 INFO  (qtp727270488-15946) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=baseConfigSet1&name=configSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078212 ERROR (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler ConfigSet: configSet1 operation: delete failed:org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:351)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.deleteConfigSet(OverseerConfigSetMessageHandler.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerConfigSetMessageHandler.processMessage(OverseerConfigSetMessageHandler.java:115)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:434)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2078212 INFO  (OverseerThreadFactory-6538-thread-5-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000046 complete, response:{Operation delete caused exception:=org.apache.solr.common.SolrException: ConfigSet does not exist to delete: configSet1,exception={msg=ConfigSet does not exist to delete: configSet1,rspCode=400}}
   [junit4]   2> 2078217 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/schema.xml to /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078218 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/schema.xml
   [junit4]   2> 2078218 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000046 state SyncConnected
   [junit4]   2> 2078221 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=configSet1&action=DELETE&wt=javabin&version=2} status=400 QTime=26 
   [junit4]   2> 2078223 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.ZkConfigManager Copying zk node /configs/grandBaseConfigSet1/solrconfig.xml to /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078224 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.c.SolrZkClient makePath: /configs/baseConfigSet1/solrconfig.xml
   [junit4]   2> 2078225 INFO  (qtp727270488-15944) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=configSet1&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 2078228 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000044 complete, response:{}
   [junit4]   2> 2078231 INFO  (OverseerCollectionConfigSetProcessor-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000048 message:{
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078231 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete , {
   [junit4]   2>   "name":"baseConfigSet1",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 2078234 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 2078235 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000044 state SyncConnected
   [junit4]   2> 2078238 INFO  (qtp727270488-15941) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2} status=0 QTime=56 
   [junit4]   2> 2078240 INFO  (OverseerThreadFactory-6538-thread-4-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskProcessor Overseer ConfigSet Message Handler: Message id:/overseer/collection-queue-work/qn-0000000048 complete, response:{}
   [junit4]   2> 2078240 INFO  (qtp727270488-15942) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :create with params baseConfigSet=grandBaseConfigSet1&name=baseConfigSet1&action=CREATE&wt=javabin&version=2 
   [junit4]   2> 2078247 INFO  (zkCallback-1893-thread-1-processing-n:127.0.0.1:50507_solr) [n:127.0.0.1:50507_solr    ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000048 state SyncConnected
   [junit4]   2> 2078250 INFO  (qtp727270488-15945) [n:127.0.0.1:50507_solr    ] o.a.s.s.SolrDispatchFilter [admin] webapp=null path=/admin/configs params={name=baseConfigSet1&action=DELETE&wt=javabin&version=2} status=0 QTime=47 
   [junit4]   2> 2078252 INFO  (qtp727270488-15943) [n:127.0.0.1:50507_solr    ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with p

[...truncated too long message...]

ce=924023624
   [junit4]   2> 2378626 INFO  (jetty-launcher-1888-thread-2) [    ] o.a.s.c.Overseer Overseer (id=94516316836986883-127.0.0.1:50507_solr-n_0000000000) closing
   [junit4]   2> 2378626 INFO  (OverseerStateUpdate-94516316836986883-127.0.0.1:50507_solr-n_0000000000) [n:127.0.0.1:50507_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50507_solr
   [junit4]   2> 2378630 INFO  (jetty-launcher-1888-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@173cdfde{/solr,null,UNAVAILABLE}
   [junit4]   2> 2378633 ERROR (jetty-launcher-1888-thread-1-EventThread) [n:127.0.0.1:50507_solr    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@a0b3888 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4112d798[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 119]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:261)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 2378635 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50396 50396
   [junit4]   2> 2378739 INFO  (Thread-7134) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:50396 50396
   [junit4]   2> 2378740 INFO  (TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[72ED523DC3CC4F54]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testAPIExclusivity
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestConfigSetsAPIExclusivity -Dtests.method=testAPIExclusivity -Dtests.seed=72ED523DC3CC4F54 -Dtests.slow=true -Dtests.locale=es_EC -Dtests.timezone=Etc/GMT+4 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE  301s J1 | TestConfigSetsAPIExclusivity.testAPIExclusivity <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Unexpected exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50507/solr: delete the configset time out:300s expected:<0> but was:<1>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72ED523DC3CC4F54:F984BC69A68F61D]:0)
   [junit4]    > 	at org.apache.solr.cloud.TestConfigSetsAPIExclusivity.testAPIExclusivity(TestConfigSetsAPIExclusivity.java:95)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2378790 INFO  (SUITE-TestConfigSetsAPIExclusivity-seed#[72ED523DC3CC4F54]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_72ED523DC3CC4F54-001
   [junit4]   2> Sep 14, 2015 4:27:22 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=DefaultSimilarity, locale=es_EC, timezone=Etc/GMT+4
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_60 (64-bit)/cpus=3,threads=1,free=240214312,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestPHPSerializedResponseWriter, TestQueryUtils, TestRecovery, DistribCursorPagingTest, TestSchemaNameResource, LeaderElectionIntegrationTest, TestComplexPhraseQParserPlugin, DistributedQueueTest, SolrCmdDistributorTest, PKIAuthenticationIntegrationTest, EchoParamsTest, TestSolrIndexConfig, BigEndianAscendingWordSerializerTest, DeleteInactiveReplicaTest, TestSchemaResource, TestInitQParser, TestRangeQuery, TestWriterPerf, TestTrie, TestSolrConfigHandlerCloud, ZkStateReaderTest, NoCacheHeaderTest, DistanceUnitsTest, TestSortingResponseWriter, MultiThreadedOCPTest, TestSolrQueryParserDefaultOperatorResource, TestAtomicUpdateErrorCases, TestCloudPivotFacet, TestOverriddenPrefixQueryForCustomFieldType, TestDistribDocBasedVersion, OutputWriterTest, TestCoreDiscovery, ResponseLogComponentTest, TestConfigSetImmutable, TestFoldingMultitermQuery, TestConfigSetsAPI, SolrPluginUtilsTest, TestMissingGroups, SpellingQueryConverterTest, TestFieldResource, TestImplicitCoreProperties, RequestHandlersTest, CursorPagingTest, RemoteQueryErrorTest, UniqFieldsUpdateProcessorFactoryTest, TestIndexSearcher, DirectUpdateHandlerOptimizeTest, TestRecoveryHdfs, DistributedFacetPivotLongTailTest, TestBadConfig, ZkStateWriterTest, TestCSVResponseWriter, AliasIntegrationTest, TestReRankQParserPlugin, ParsingFieldUpdateProcessorsTest, BinaryUpdateRequestHandlerTest, TestSolrQueryParser, TestSolrCLIRunExample, TestPKIAuthenticationPlugin, TimeZoneUtilsTest, CollectionTooManyReplicasTest, TestValueSourceCache, TestMiniSolrCloudClusterSSL, BasicDistributedZk2Test, UpdateRequestProcessorFactoryTest, TestStressUserVersions, TestStressRecovery, TestConfigSetsAPIZkFailure, ChaosMonkeySafeLeaderTest, HighlighterConfigTest, AddSchemaFieldsUpdateProcessorFactoryTest, RequestLoggingTest, TestTolerantSearch, TestHighFrequencyDictionaryFactory, BasicFunctionalityTest, TestBM25SimilarityFactory, TestJsonFacets, TestAuthorizationFramework, CSVRequestHandlerTest, FullSolrCloudDistribCmdsTest, BlockDirectoryTest, MinimalSchemaTest, SuggesterTest, TestTrieFacet, TestUniqueKeyFieldResource, CircularListTest, TestIndexingPerformance, VersionInfoTest, HighlighterMaxOffsetTest, TestInfoStreamLogging, TestLMJelinekMercerSimilarityFactory, ShardRoutingCustomTest, ConvertedLegacyTest, AssignTest, TestDistributedGrouping, TestDynamicFieldResource, TestManagedSchemaFieldTypeResource, TestSolrDeletionPolicy2, AsyncCallRequestStatusResponseTest, TestChildDocTransformer, SuggesterWFSTTest, TestFuzzyAnalyzedSuggestions, TestMiniSolrCloudClusterBase, ResponseHeaderTest, AnalyticsQueryTest, TestJoin, SolrTestCaseJ4Test, DistributedFacetPivotSmallAdvancedTest, TestMacros, TestLeaderElectionZkExpiry, DistanceFunctionTest, TestReload, TestSolrCoreProperties, TestDynamicFieldCollectionResource, EnumFieldTest, TestFieldTypeCollectionResource, BufferStoreTest, ShowFileRequestHandlerTest, TestManagedSchemaFieldResource, TestCharFilters, SolrInfoMBeanTest, TestDefaultSimilarityFactory, SimpleMLTQParserTest, TestFiltering, TestConfigOverlay, HdfsNNFailoverTest, DateRangeFieldTest, TriLevelCompositeIdRoutingTest, TestBlendedInfixSuggestions, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, BasicZkTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, AutoCommitTest, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestRealTimeGet, TestStressVersions, DistributedTermsComponentTest, TestSolr4Spatial, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestFunctionQuery, TestLazyCores, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, CoreAdminHandlerTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, TestQueryTypes, TestOmitPositions, XmlUpdateRequestHandlerTest, TermVectorComponentTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, IndexSchemaTest, QueryParsingTest, PingRequestHandlerTest, NotRequiredUniqueKeyTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestJmxMonitoredMap, ScriptEngineTest, PluginInfoTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, CollectionStateFormat2Test, DistribJoinFromCollectionTest, HttpPartitionTest, OverseerStatusTest, OverseerTaskQueueTest, RollingRestartTest, TestCloudInspectUtil, TestConfigSetsAPIExclusivity]
   [junit4] Completed [431/538] on J1 in 301.71s, 1 test, 1 failure <<< FAILURES!

[...truncated 351 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:729: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:524: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1452: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1006: There were test failures: 538 suites, 2127 tests, 1 failure, 46 ignored (34 assumptions)

Total time: 72 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 50 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/50/
Java: multiarch/jdk1.7.0 -d32 -client -XX:+UseG1GC

38 tests failed.
FAILED:  org.apache.solr.cloud.AliasIntegrationTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.AsyncCallRequestStatusResponseTest.testAsyncCallStatusResponse

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:5A011F46FA529EFD]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
Unable to create core [testcore]

Stack Trace:
org.apache.solr.common.SolrException: Unable to create core [testcore]
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:BE8B1C05AD121D11]:0)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:741)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:701)
	at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:163)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper:conf1
	at org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:164)
	at org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:37)
	at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:71)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:725)
	... 41 more


FAILED:  org.apache.solr.cloud.CollectionReloadTest.testReloadedLeaderStateAfterZkSessionLoss

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:FB2EC1F772856071]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionStateFormat2Test.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddShard

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:24451C0B8C7DCD6F]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testAddTooManyReplicas

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:8D520E729A8E2C95]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionTooManyReplicasTest.testDownedShards

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:6228D2FF5F935E60]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteShardTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistribJoinFromCollectionTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistributedVersionInfoTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.MigrateRouteKeyTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.OverseerStatusTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
	at org.apache.solr.cloud.RecoveryZkTest.distribTearDown(RecoveryZkTest.java:154)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
55 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=4294, name=qtp26618513-4294, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=4323, name=Scheduler-24288794, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=4333, name=zkCallback-490-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=4300, name=org.eclipse.jetty.server.session.HashSessionManager@26d9a3Timer, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=4282, name=OverseerHdfsCoreFailoverThread-94514483894812676-127.0.0.1:50062_-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=4270, name=org.eclipse.jetty.server.session.HashSessionManager@1634769Timer, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=4281, name=OverseerCollectionConfigSetProcessor-94514483894812676-127.0.0.1:50062_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:344)         at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:210)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:197)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=4337, name=coreContainerWorkExecutor-1690-thread-1, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=4253, name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)   10) Thread[id=4324, name=org.eclipse.jetty.server.session.HashSessionManager@78540Timer, state=TIMED_WAITING, group=TGRP-RecoveryZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  

[...truncated too long message...]

)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:380)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:377)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:377)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:948)
   [junit4]   2> 	... 13 more
   [junit4]   2> 995179 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@1e9deec{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 995180 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@112528d{/_vic/e,null,UNAVAILABLE}
   [junit4]   2> 995181 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 41422
   [junit4]   2> 995181 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2421242
   [junit4]   2> 995182 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.a.s.c.Overseer Overseer (id=94514495826821134-127.0.0.1:41422__vic%2Fe-n_0000000003) closing
   [junit4]   2> 995183 INFO  (OverseerStateUpdate-94514495826821134-127.0.0.1:41422__vic%2Fe-n_0000000003) [n:127.0.0.1:41422__vic%2Fe    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41422__vic%2Fe
   [junit4]   2> 995183 INFO  (zkCallback-839-thread-1-processing-n:127.0.0.1:41422__vic%2Fe) [n:127.0.0.1:41422__vic%2Fe    ] o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 995184 WARN  (zkCallback-839-thread-1-processing-n:127.0.0.1:41422__vic%2Fe) [n:127.0.0.1:41422__vic%2Fe    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 995186 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@ea1e26{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 995186 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@17b6eb{/_vic/e,null,UNAVAILABLE}
   [junit4]   2> 995189 INFO  (TEST-TriLevelCompositeIdRoutingTest.test-seed#[7AAAD4681324]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58809 58809
   [junit4]   2> 995299 INFO  (Thread-3700) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58809 58809
   [junit4]   2> 995300 WARN  (Thread-3700) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/aliases.json
   [junit4]   2> 	4	/solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 	5	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 	2	/solr/overseer/queue
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.method=test -Dtests.seed=7AAAD4681324 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=Pacific/Wake -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   21.5s J1 | TriLevelCompositeIdRoutingTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7AAAD4681324:885445707A947EDC]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:642)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:405)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:310)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:961)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 995303 INFO  (SUITE-TriLevelCompositeIdRoutingTest-seed#[7AAAD4681324]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TriLevelCompositeIdRoutingTest_7AAAD4681324-001
   [junit4]   2> Sep 13, 2015 8:39:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene53): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=sr_CS, timezone=Pacific/Wake
   [junit4]   2> NOTE: SunOS 5.11 x86/Oracle Corporation 1.7.0_85 (32-bit)/cpus=3,threads=1,free=214414184,total=407896064
   [junit4]   2> NOTE: All tests run in this JVM: [BufferStoreTest, TestObjectReleaseTracker, TestConfigSetsAPI, CollectionReloadTest, TestReversedWildcardFilterFactory, StressHdfsTest, TestSha256AuthenticationProvider, BlockDirectoryTest, TestManagedSchemaFieldTypeResource, TestHdfsUpdateLog, TestDistribIDF, TestCoreDiscovery, TestFoldingMultitermQuery, TestSolrQueryParserResource, SolrInfoMBeanTest, TestQuerySenderListener, LukeRequestHandlerTest, CircularListTest, CloudMLTQParserTest, TestExpandComponent, TestPerFieldSimilarity, CollectionStateFormat2Test, AliasIntegrationTest, BigEndianAscendingWordSerializerTest, TestIndexingPerformance, TestReqParamsAPI, QueryParsingTest, OverriddenZkACLAndCredentialsProvidersTest, TestBlobHandler, TestRecovery, TestCustomDocTransformer, TestTolerantSearch, TestInitParams, SuggesterWFSTTest, NumericFieldsTest, TestExactSharedStatsCache, SolrCoreCheckLockOnStartupTest, AsyncCallRequestStatusResponseTest, TestLeaderInitiatedRecoveryThread, DebugComponentTest, ChangedSchemaMergeTest, TestCSVResponseWriter, DocValuesMissingTest, DistributedIntervalFacetingTest, TestQueryUtils, TestValueSourceCache, DistributedVersionInfoTest, CurrencyFieldOpenExchangeTest, TestSolrConfigHandlerCloud, ConvertedLegacyTest, TestRTimerTree, TestFiltering, CursorPagingTest, TestBinaryField, SliceStateTest, CloudExitableDirectoryReaderTest, MigrateRouteKeyTest, AutoCommitTest, BinaryUpdateRequestHandlerTest, SolrCloudExampleTest, CoreMergeIndexesAdminHandlerTest, RollingRestartTest, UpdateRequestProcessorFactoryTest, SuggesterTest, NotRequiredUniqueKeyTest, DistanceFunctionTest, FieldAnalysisRequestHandlerTest, AtomicUpdatesTest, TestSolrCloudWithKerberosAlt, BitVectorTest, TestSchemaNameResource, TestManagedResourceStorage, SolrTestCaseJ4Test, TestDistribDocBasedVersion, ZkStateWriterTest, BJQParserTest, TestUniqueKeyFieldResource, LeaderInitiatedRecoveryOnCommitTest, TestFieldTypeResource, OutOfBoxZkACLAndCredentialsProvidersTest, DistributedFacetPivotSmallAdvancedTest, TestSchemaManager, DirectUpdateHandlerOptimizeTest, OutputWriterTest, DistributedExpandComponentTest, TestHighlightDedupGrouping, TestSimpleQParserPlugin, SuggesterTSTTest, RankQueryTest, TestReplicationHandlerBackup, TestNoOpRegenerator, TestFreeTextSuggestions, OverseerStatusTest, TestIntervalFaceting, ZkStateReaderTest, BasicFunctionalityTest, URLClassifyProcessorTest, TestSolrIndexConfig, HdfsThreadLeakTest, FullHLLTest, PreAnalyzedFieldTest, ResponseLogComponentTest, OpenCloseCoreStressTest, DeleteShardTest, TimeZoneUtilsTest, SecurityConfHandlerTest, TestCryptoKeys, TestSolrConfigHandlerConcurrent, RequestLoggingTest, TestStressRecovery, LoggingHandlerTest, TestConfigOverlay, HttpPartitionTest, TermVectorComponentDistributedTest, TestDefaultSimilarityFactory, DistributedQueueTest, HdfsUnloadDistributedZkTest, TestJoin, TestShortCircuitedRequests, TriLevelCompositeIdRoutingTest]
   [junit4] Completed [525/538] on J1 in 22.01s, 1 test, 1 error <<< FAILURES!

[...truncated 58 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:729: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/common-build.xml:524: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1452: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/lucene/common-build.xml:1006: There were test failures: 538 suites, 2127 tests, 2 suite-level errors, 36 errors, 1333 ignored (26 assumptions)

Total time: 38 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any