You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/13 03:21:37 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_20-ea-b21) - Build # 10680 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10680/
Java: 32bit/jdk1.8.0_20-ea-b21 -client -XX:+UseParallelGC

7 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ReplicationFactorTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE]:0)


REGRESSION:  org.apache.solr.cloud.CustomCollectionTest.testDistribSearch

Error Message:
expected:<0> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<3>
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE:E9D589202491EEF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:248)
	at org.apache.solr.cloud.CustomCollectionTest.doTest(CustomCollectionTest.java:127)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
shard2 is not consistent.  Got 1 from http://127.0.0.1:43699/_oq/collection1lastClient and got 496 from http://127.0.0.1:59145/_oq/collection1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 1 from http://127.0.0.1:43699/_oq/collection1lastClient and got 496 from http://127.0.0.1:59145/_oq/collection1
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE:E9D589202491EEF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1110)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingDocPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:333)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:134)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"state":"active","base_url":"http://127.0.0.1:60305/_lv/h","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:60305__lv%2Fh","leader":"true"}]; clusterState: live nodes:[127.0.0.1:50175__lv%2Fh, 127.0.0.1:60305__lv%2Fh] collections:{control_collection=DocCollection(control_collection)={   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:60305/_lv/h",           "core":"collection1",           "node_name":"127.0.0.1:60305__lv%2Fh",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{"core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:50175/_lv/h",           "core":"collection1",           "node_name":"127.0.0.1:50175__lv%2Fh",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:38372/_lv/h",           "core":"collection1",           "node_name":"127.0.0.1:38372__lv%2Fh",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:60305/_lv/h",           "core":"c8n_1x2_shard1_replica2",           "node_name":"127.0.0.1:60305__lv%2Fh",           "leader":"true"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:38372/_lv/h",           "core":"c8n_1x2_shard1_replica1",           "node_name":"127.0.0.1:38372__lv%2Fh"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:60305/_lv/h",           "core":"c8n_1x3_shard1_replica2",           "node_name":"127.0.0.1:60305__lv%2Fh"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:50175/_lv/h",           "core":"c8n_1x3_shard1_replica1",           "node_name":"127.0.0.1:50175__lv%2Fh",           "leader":"true"},         "core_node3":{           "state":"down",           "base_url":"http://127.0.0.1:38372/_lv/h",           "core":"c8n_1x3_shard1_replica3",           "node_name":"127.0.0.1:38372__lv%2Fh"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:38372/_lv/h",           "core":"c8n_1x3_lf_shard1_replica2",           "node_name":"127.0.0.1:38372__lv%2Fh"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:50175/_lv/h",           "core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:50175__lv%2Fh"},         "core_node3":{           "state":"active",           "base_url":"http://127.0.0.1:60305/_lv/h",           "core":"c8n_1x3_lf_shard1_replica3",           "node_name":"127.0.0.1:60305__lv%2Fh",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"state":"active","base_url":"http://127.0.0.1:60305/_lv/h","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:60305__lv%2Fh","leader":"true"}]; clusterState: live nodes:[127.0.0.1:50175__lv%2Fh, 127.0.0.1:60305__lv%2Fh] collections:{control_collection=DocCollection(control_collection)={
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:60305/_lv/h",
          "core":"collection1",
          "node_name":"127.0.0.1:60305__lv%2Fh",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{"core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:50175/_lv/h",
          "core":"collection1",
          "node_name":"127.0.0.1:50175__lv%2Fh",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:38372/_lv/h",
          "core":"collection1",
          "node_name":"127.0.0.1:38372__lv%2Fh",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:60305/_lv/h",
          "core":"c8n_1x2_shard1_replica2",
          "node_name":"127.0.0.1:60305__lv%2Fh",
          "leader":"true"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:38372/_lv/h",
          "core":"c8n_1x2_shard1_replica1",
          "node_name":"127.0.0.1:38372__lv%2Fh"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:60305/_lv/h",
          "core":"c8n_1x3_shard1_replica2",
          "node_name":"127.0.0.1:60305__lv%2Fh"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:50175/_lv/h",
          "core":"c8n_1x3_shard1_replica1",
          "node_name":"127.0.0.1:50175__lv%2Fh",
          "leader":"true"},
        "core_node3":{
          "state":"down",
          "base_url":"http://127.0.0.1:38372/_lv/h",
          "core":"c8n_1x3_shard1_replica3",
          "node_name":"127.0.0.1:38372__lv%2Fh"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:38372/_lv/h",
          "core":"c8n_1x3_lf_shard1_replica2",
          "node_name":"127.0.0.1:38372__lv%2Fh"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:50175/_lv/h",
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:50175__lv%2Fh"},
        "core_node3":{
          "state":"active",
          "base_url":"http://127.0.0.1:60305/_lv/h",
          "core":"c8n_1x3_lf_shard1_replica3",
          "node_name":"127.0.0.1:60305__lv%2Fh",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}}
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE:E9D589202491EEF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:368)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.ReplicationFactorTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE]:0)


REGRESSION:  org.apache.solr.cloud.ShardRoutingTest.testDistribSearch

Error Message:
expected:<[e!doc4, b!doc1, d!doc3, f1!f2!doc5]> but was:<[e!doc4, b!doc1, c!doc2, d!doc3, f1!f2!doc5]>

Stack Trace:
java.lang.AssertionError: expected:<[e!doc4, b!doc1, d!doc3, f1!f2!doc5]> but was:<[e!doc4, b!doc1, c!doc2, d!doc3, f1!f2!doc5]>
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE:E9D589202491EEF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.doQuery(AbstractFullDistribZkTestBase.java:1414)
	at org.apache.solr.cloud.ShardRoutingTest.doHashingTest(ShardRoutingTest.java:207)
	at org.apache.solr.cloud.ShardRoutingTest.doTest(ShardRoutingTest.java:123)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 4 from https://127.0.0.1:44186/yz/t/collection1lastClient and got 2 from https://127.0.0.1:37029/yz/t/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 4 from https://127.0.0.1:44186/yz/t/collection1lastClient and got 2 from https://127.0.0.1:37029/yz/t/collection1
	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE:E9D589202491EEF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:145)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11628 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/init-core-data-001
   [junit4]   2> 585511 T2790 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_lv/h
   [junit4]   2> 585517 T2790 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 585517 T2790 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 585518 T2791 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 585618 T2790 oasc.ZkTestServer.run start zk server on port:53817
   [junit4]   2> 585619 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 585621 T2797 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127acf0 name:ZooKeeperConnection Watcher:127.0.0.1:53817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 585622 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 585622 T2790 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 585625 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 585626 T2799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f3e38b name:ZooKeeperConnection Watcher:127.0.0.1:53817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 585626 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 585627 T2790 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 585629 T2790 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 585631 T2790 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 585633 T2790 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 585634 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 585635 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 585638 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 585639 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 585641 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 585642 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 585644 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 585645 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 585647 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 585648 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 585650 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 585651 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 585652 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 585653 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 585655 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 585655 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 585659 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 585659 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 585661 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 585662 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 585664 T2790 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 585665 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 585958 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 585965 T2790 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:32866
   [junit4]   2> 585966 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 585966 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 585966 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002
   [junit4]   2> 585967 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/'
   [junit4]   2> 585992 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/solr.xml
   [junit4]   2> 586010 T2790 oasc.CoreContainer.<init> New CoreContainer 1805125
   [junit4]   2> 586011 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/]
   [junit4]   2> 586012 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 586012 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 586012 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 586013 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 586013 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 586013 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 586014 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 586014 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 586014 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 586015 T2790 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 586016 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 586016 T2790 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 586017 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53817/solr
   [junit4]   2> 586017 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 586018 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 586020 T2810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bd6627 name:ZooKeeperConnection Watcher:127.0.0.1:53817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 586020 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 586022 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 586023 T2812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81b3f2 name:ZooKeeperConnection Watcher:127.0.0.1:53817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 586023 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 586024 T2790 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 586026 T2790 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 586028 T2790 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 586030 T2790 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 586032 T2790 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 586035 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 586036 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60305__lv%2Fh
   [junit4]   2> 586037 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60305__lv%2Fh
   [junit4]   2> 586039 T2790 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 586040 T2790 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 586042 T2790 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60305__lv%2Fh
   [junit4]   2> 586043 T2790 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 586045 T2790 oasc.Overseer.start Overseer (id=92091655120617475-127.0.0.1:60305__lv%2Fh-n_0000000000) starting
   [junit4]   2> 586047 T2790 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 586053 T2814 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 586054 T2790 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 586056 T2790 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 586057 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 586059 T2813 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 586062 T2815 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 586062 T2815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 586063 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 586063 T2815 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 586064 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 586065 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60305/_lv/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60305__lv%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 586065 T2813 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 586065 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 586067 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 586067 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 587064 T2815 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 587064 T2815 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 587065 T2815 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 587065 T2815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 587066 T2815 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 587066 T2815 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/collection1/'
   [junit4]   2> 587067 T2815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 587067 T2815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 587068 T2815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 587113 T2815 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 587125 T2815 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 587126 T2815 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 587144 T2815 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 587274 T2815 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 587274 T2815 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 587275 T2815 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 587280 T2815 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 587282 T2815 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 587296 T2815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 587299 T2815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 587301 T2815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 587302 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 587302 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 587303 T2815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 587303 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 587304 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 587304 T2815 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 587304 T2815 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 587304 T2815 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/control/data/
   [junit4]   2> 587305 T2815 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f892b7
   [junit4]   2> 587305 T2815 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/control/data
   [junit4]   2> 587305 T2815 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/control/data/index/
   [junit4]   2> 587306 T2815 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 587306 T2815 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/control/data/index
   [junit4]   2> 587306 T2815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=15.1767578125, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=14.503553684610699, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 587307 T2815 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@238209 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ebae4a),segFN=segments_1,generation=1}
   [junit4]   2> 587307 T2815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 587310 T2815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 587310 T2815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 587310 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 587311 T2815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 587311 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 587311 T2815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 587311 T2815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 587312 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 587312 T2815 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 587312 T2815 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 587313 T2815 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 587313 T2815 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 587313 T2815 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 587314 T2815 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 587314 T2815 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 587315 T2815 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 587316 T2815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 587318 T2815 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 587318 T2815 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 587319 T2815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3121865003345368]
   [junit4]   2> 587319 T2815 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@238209 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ebae4a),segFN=segments_1,generation=1}
   [junit4]   2> 587320 T2815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 587320 T2815 oass.SolrIndexSearcher.<init> Opening Searcher@1f020cb[collection1] main
   [junit4]   2> 587320 T2815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 587321 T2815 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 587321 T2815 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 587322 T2815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 587322 T2815 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 587322 T2815 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 587323 T2815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 587323 T2815 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 587323 T2815 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 587323 T2815 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 587326 T2816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f020cb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 587327 T2815 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 587328 T2819 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60305/_lv/h collection:control_collection shard:shard1
   [junit4]   2> 587328 T2790 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 587329 T2819 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 587329 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 587332 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 587332 T2819 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 587333 T2822 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bbaa43 name:ZooKeeperConnection Watcher:127.0.0.1:53817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 587333 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 587334 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587334 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 587334 T2819 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 587335 T2819 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C937 name=collection1 org.apache.solr.core.SolrCore@9ecdde url=http://127.0.0.1:60305/_lv/h/collection1 node=127.0.0.1:60305__lv%2Fh C937_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:60305/_lv/h, core=collection1, node_name=127.0.0.1:60305__lv%2Fh}
   [junit4]   2> 587335 T2819 C937 P60305 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60305/_lv/h/collection1/
   [junit4]   2> 587335 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 587335 T2819 C937 P60305 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 587336 T2819 C937 P60305 oasc.SyncStrategy.syncToMe http://127.0.0.1:60305/_lv/h/collection1/ has no replicas
   [junit4]   2> 587336 T2819 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60305/_lv/h/collection1/ shard1
   [junit4]   2> 587336 T2819 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 587336 T2790 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 587337 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587339 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587342 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587443 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 587443 T2822 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 587490 T2819 oasc.ZkController.register We are http://127.0.0.1:60305/_lv/h/collection1/ and leader is http://127.0.0.1:60305/_lv/h/collection1/
   [junit4]   2> 587491 T2819 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60305/_lv/h
   [junit4]   2> 587491 T2819 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 587491 T2819 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 587492 T2819 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 587493 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587493 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587494 T2819 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 587494 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587495 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 587496 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60305/_lv/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60305__lv%2Fh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 587497 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 587599 T2822 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 587599 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 587622 T2790 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 587623 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 587625 T2790 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45531
   [junit4]   2> 587626 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 587626 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 587626 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003
   [junit4]   2> 587627 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/'
   [junit4]   2> 587646 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/solr.xml
   [junit4]   2> 587661 T2790 oasc.CoreContainer.<init> New CoreContainer 31993077
   [junit4]   2> 587662 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/]
   [junit4]   2> 587662 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 587663 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 587663 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 587664 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 587664 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 587664 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 587665 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 587665 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 587665 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 587666 T2790 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 587666 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 587666 T2790 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 587667 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53817/solr
   [junit4]   2> 587667 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 587668 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 587669 T2833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a64d6 name:ZooKeeperConnection Watcher:127.0.0.1:53817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 587670 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 587671 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 587672 T2835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a831f7 name:ZooKeeperConnection Watcher:127.0.0.1:53817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 587672 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 587677 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 588680 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38372__lv%2Fh
   [junit4]   2> 588681 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38372__lv%2Fh
   [junit4]   2> 588683 T2812 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 588683 T2822 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 588683 T2835 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 588688 T2836 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 588688 T2836 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 588689 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 588689 T2836 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 588689 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 588690 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 588690 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 588690 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38372/_lv/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38372__lv%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 588691 T2813 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 588691 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 588693 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 588795 T2822 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 588795 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 588795 T2835 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 589689 T2836 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 589690 T2836 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 589691 T2836 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 589691 T2836 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 589692 T2836 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 589692 T2836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/collection1/'
   [junit4]   2> 589693 T2836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 589694 T2836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 589695 T2836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 589729 T2836 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 589742 T2836 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 589743 T2836 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 589754 T2836 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 589894 T2836 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 589895 T2836 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 589895 T2836 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 589900 T2836 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 589902 T2836 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 589918 T2836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 589921 T2836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 589924 T2836 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 589925 T2836 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 589925 T2836 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 589926 T2836 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 589927 T2836 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 589927 T2836 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 589927 T2836 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 589927 T2836 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 589928 T2836 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/jetty1/
   [junit4]   2> 589928 T2836 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f892b7
   [junit4]   2> 589929 T2836 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/jetty1
   [junit4]   2> 589929 T2836 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/jetty1/index/
   [junit4]   2> 589929 T2836 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 589929 T2836 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-001/jetty1/index
   [junit4]   2> 589930 T2836 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=15.1767578125, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=14.503553684610699, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 589930 T2836 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d08fd2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2ccd6),segFN=segments_1,generation=1}
   [junit4]   2> 589931 T2836 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 589934 T2836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 589934 T2836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 589935 T2836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 589935 T2836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 589935 T2836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 589935 T2836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 589936 T2836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 589936 T2836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 589936 T2836 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 589937 T2836 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 589938 T2836 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 589939 T2836 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 589939 T2836 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 589940 T2836 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 589941 T2836 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 589941 T2836 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 589944 T2836 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 589945 T2836 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 589945 T2836 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 589946 T2836 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3121865003345368]
   [junit4]   2> 589947 T2836 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d08fd2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a2ccd6),segFN=segments_1,generation=1}
   [junit4]   2> 589947 T2836 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 589947 T2836 oass.SolrIndexSearcher.<init> Opening Searcher@5f3717[collection1] main
   [junit4]   2> 589948 T2836 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 589949 T2836 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 589949 T2836 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 589949 T2836 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 589950 T2836 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 589950 T2836 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 589951 T2836 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 589951 T2836 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 589951 T2836 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 589951 T2836 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 589956 T2837 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f3717[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 589959 T2836 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 589959 T2840 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38372/_lv/h collection:collection1 shard:shard2
   [junit4]   2> 589960 T2790 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 589960 T2840 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 589960 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 589964 T2840 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 589966 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 589966 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 589966 T2840 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 589967 T2840 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 589967 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 589967 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C938 name=collection1 org.apache.solr.core.SolrCore@160a80c url=http://127.0.0.1:38372/_lv/h/collection1 node=127.0.0.1:38372__lv%2Fh C938_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:38372/_lv/h, core=collection1, node_name=127.0.0.1:38372__lv%2Fh}
   [junit4]   2> 589967 T2840 C938 P38372 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38372/_lv/h/collection1/
   [junit4]   2> 589968 T2840 C938 P38372 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 589968 T2840 C938 P38372 oasc.SyncStrategy.syncToMe http://127.0.0.1:38372/_lv/h/collection1/ has no replicas
   [junit4]   2> 589969 T2840 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38372/_lv/h/collection1/ shard2
   [junit4]   2> 589969 T2840 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 589969 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 589973 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 589975 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 590077 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590077 T2835 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590077 T2822 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590125 T2840 oasc.ZkController.register We are http://127.0.0.1:38372/_lv/h/collection1/ and leader is http://127.0.0.1:38372/_lv/h/collection1/
   [junit4]   2> 590125 T2840 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38372/_lv/h
   [junit4]   2> 590125 T2840 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 590126 T2840 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 590126 T2840 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 590127 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 590127 T2840 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 590127 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 590128 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 590129 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 590130 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38372/_lv/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38372__lv%2Fh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 590132 T2812 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 590234 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590234 T2822 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590234 T2835 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 590241 T2790 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 590242 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 590244 T2790 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58945
   [junit4]   2> 590246 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 590246 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 590247 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-004
   [junit4]   2> 590247 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-004/'
   [junit4]   2> 590267 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-004/solr.xml
   [junit4]   2> 590289 T2790 oasc.CoreContainer.<init> New CoreContainer 26036619
   [junit4]   2> 590290 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6833073853CE8ECE-001/tempDir-004/]
   [junit4]   2> 590291 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 590291 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 590291 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 590292 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 590292 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 590292 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 590293 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 590293 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 590293 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 590294 T2790 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 590294 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 590294 T2790 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 590295 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53817/solr
   [junit4]   2> 590295 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 590296 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 590297 T2852 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2c6e0 name:ZooKeeperConnection Watcher:127.0.0.1:53817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 590297 T27

[...truncated too long message...]

5)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer$ZKServerMain.shutdown(ZkTestServer.java:126)
   [junit4]   2> 	        at org.apache.solr.cloud.ZkTestServer.shutdown(ZkTestServer.java:262)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:232)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1435)
   [junit4]   2> 	        at org.apache.solr.cloud.ReplicationFactorTest.tearDown(ReplicationFactorTest.java:88)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:483)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 7920422 T3093 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=lv_LV, timezone=America/Buenos_Aires
   [junit4]   2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.8.0_20-ea (32-bit)/cpus=8,threads=2,free=147033792,total=273416192
   [junit4]   2> NOTE: All tests run in this JVM: [SpellCheckCollatorTest, TestUtils, AssignTest, TestConfig, TestSchemaNameResource, EchoParamsTest, TestCloudManagedSchemaConcurrent, TestCollationFieldDocValues, TestBM25SimilarityFactory, DisMaxRequestHandlerTest, TestDocSet, TestHashPartitioner, FieldMutatingUpdateProcessorTest, TestDefaultSimilarityFactory, DocumentAnalysisRequestHandlerTest, MBeansHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestCursorMarkWithoutUniqueKey, ReturnFieldsTest, TestPartialUpdateDeduplication, HighlighterMaxOffsetTest, CurrencyFieldOpenExchangeTest, TermsComponentTest, TestSolr4Spatial2, TestSolrXMLSerializer, RegexBoostProcessorTest, BJQParserTest, TestDFRSimilarityFactory, CoreAdminCreateDiscoverTest, IndexBasedSpellCheckerTest, TestDynamicFieldCollectionResource, TestSearcherReuse, TestReplicationHandlerBackup, TestManagedSchema, PeerSyncTest, TestCopyFieldCollectionResource, CoreMergeIndexesAdminHandlerTest, TestHighFrequencyDictionaryFactory, TestSort, TestShardHandlerFactory, TestFieldTypeCollectionResource, HdfsRecoveryZkTest, SolrCoreCheckLockOnStartupTest, TestFieldCollectionResource, TestRequestStatusCollectionAPI, RAMDirectoryFactoryTest, ResponseLogComponentTest, TestMiniSolrCloudCluster, IndexSchemaTest, HdfsCollectionsAPIDistributedZkTest, TestStressReorder, TestBlendedInfixSuggestions, ResourceLoaderTest, TestJmxIntegration, ClusterStateUpdateTest, OverseerCollectionProcessorTest, TestStressRecovery, NoCacheHeaderTest, HdfsLockFactoryTest, TestNRTOpen, TestDistributedMissingSort, TestSolrIndexConfig, XmlUpdateRequestHandlerTest, ShowFileRequestHandlerTest, BlockDirectoryTest, DateMathParserTest, ZkCLITest, UUIDFieldTest, BasicDistributedZkTest, TestDistributedSearch, TestLuceneMatchVersion, TestStressVersions, LeaderElectionTest, HdfsDirectoryTest, BadComponentTest, PluginInfoTest, AsyncMigrateRouteKeyTest, BadCopyFieldTest, DirectUpdateHandlerTest, TestDistribDocBasedVersion, SimpleFacetsTest, DocValuesMultiTest, HighlighterConfigTest, ZkControllerTest, QueryResultKeyTest, TestFoldingMultitermQuery, TestCloudInspectUtil, TestManagedStopFilterFactory, TestHighlightDedupGrouping, UnloadDistributedZkTest, HttpPartitionTest, OpenExchangeRatesOrgProviderTest, ReplicationFactorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ReplicationFactorTest -Dtests.seed=6833073853CE8ECE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lv_LV -Dtests.timezone=America/Buenos_Aires -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | ReplicationFactorTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6833073853CE8ECE]:0)
   [junit4] Completed on J0 in 7223.51s, 1 test, 2 errors <<< FAILURES!

[...truncated 31 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 1 suite-level error, 1 error, 5 failures, 61 ignored (16 assumptions)

Total time: 174 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20-ea-b21 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_05) - Build # 10684 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10684/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard3 is not consistent.  Got 159 from https://127.0.0.1:43188/collection1lastClient and got 135 from https://127.0.0.1:37995/collection1

Stack Trace:
java.lang.AssertionError: shard3 is not consistent.  Got 159 from https://127.0.0.1:43188/collection1lastClient and got 135 from https://127.0.0.1:37995/collection1
	at __randomizedtesting.SeedInfo.seed([788BAFCA394B56F3:F96D21D24E1436CF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.RollingRestartTest.testDistribSearch

Error Message:
No overseer designate as leader found after restart #1: 127.0.0.1:38817__

Stack Trace:
java.lang.AssertionError: No overseer designate as leader found after restart #1: 127.0.0.1:38817__
	at __randomizedtesting.SeedInfo.seed([788BAFCA394B56F3:F96D21D24E1436CF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:100)
	at org.apache.solr.cloud.RollingRestartTest.doTest(RollingRestartTest.java:61)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11410 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/init-core-data-001
   [junit4]   2> 445819 T2706 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 445819 T2706 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/
   [junit4]   2> 445821 T2706 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 445822 T2706 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 445822 T2707 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 445922 T2706 oasc.ZkTestServer.run start zk server on port:38707
   [junit4]   2> 445923 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445925 T2713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77db09b7 name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445925 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 445926 T2706 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 445929 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445930 T2715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e42d1ed name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445930 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 445930 T2706 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 445931 T2706 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 445932 T2706 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 445933 T2706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 445934 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 445934 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 445936 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 445936 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 445937 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 445938 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 445939 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 445939 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 445940 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 445941 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 445942 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 445942 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 445943 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 445943 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 445944 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 445945 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 445946 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 445946 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 445947 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 445948 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 445949 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 445949 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 445951 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445952 T2717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@503237a0 name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445952 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446187 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 446190 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 446199 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43392
   [junit4]   2> 446201 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 446201 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 446202 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002
   [junit4]   2> 446202 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/'
   [junit4]   2> 446217 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/solr.xml
   [junit4]   2> 446226 T2706 oasc.CoreContainer.<init> New CoreContainer 277018058
   [junit4]   2> 446227 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/]
   [junit4]   2> 446227 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 446230 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 446230 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 446232 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 446233 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 446233 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 446234 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 446234 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 446235 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446236 T2728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ba83ecb name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446236 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446238 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446239 T2730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48c70c4d name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446240 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446241 T2706 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 446243 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 446248 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 446250 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 446252 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 446254 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 446256 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43392__
   [junit4]   2> 446256 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43392__
   [junit4]   2> 446258 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 446260 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 446262 T2706 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43392__
   [junit4]   2> 446262 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 446263 T2706 oasc.Overseer.start Overseer (id=92094321097703428-127.0.0.1:43392__-n_0000000000) starting
   [junit4]   2> 446265 T2706 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 446270 T2732 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 446271 T2706 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 446272 T2706 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 446273 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 446275 T2731 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 446276 T2733 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 446276 T2733 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 446277 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 446277 T2733 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 446278 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43392/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43392__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 446279 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 446280 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447277 T2733 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 447278 T2733 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 447278 T2733 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 447279 T2733 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 447279 T2733 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/'
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 447281 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 447297 T2733 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 447313 T2733 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 447314 T2733 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 447318 T2733 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 447429 T2733 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 447430 T2733 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 447430 T2733 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 447434 T2733 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 447435 T2733 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 447442 T2733 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 447444 T2733 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 447446 T2733 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 447448 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 447448 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 447448 T2733 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 447448 T2733 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 447448 T2733 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/
   [junit4]   2> 447449 T2733 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 447449 T2733 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data
   [junit4]   2> 447449 T2733 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index/
   [junit4]   2> 447449 T2733 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 447450 T2733 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index
   [junit4]   2> 447450 T2733 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 447520 T2733 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 447520 T2733 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 447523 T2733 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 447523 T2733 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 447525 T2733 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 447526 T2733 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 447527 T2733 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 447527 T2733 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 447527 T2733 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 447528 T2733 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 447528 T2733 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 447528 T2733 oass.SolrIndexSearcher.<init> Opening Searcher@31c6243[collection1] main
   [junit4]   2> 447529 T2733 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 447530 T2733 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 447530 T2733 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 447530 T2733 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 447530 T2733 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 447531 T2733 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 447531 T2733 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 447533 T2734 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31c6243[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 447533 T2733 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 447534 T2737 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43392/_ collection:control_collection shard:shard1
   [junit4]   2> 447534 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 447535 T2737 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 447535 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 447537 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447537 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 447538 T2739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6620eed3 name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447538 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447538 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447539 T2737 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 447539 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 447539 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 447539 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C894 name=collection1 org.apache.solr.core.SolrCore@3ab1d5d4 url=https://127.0.0.1:43392/_/collection1 node=127.0.0.1:43392__ C894_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:43392/_, core=collection1, node_name=127.0.0.1:43392__}
   [junit4]   2> 447539 T2737 C894 P43392 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43392/_/collection1/
   [junit4]   2> 447540 T2737 C894 P43392 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 447540 T2737 C894 P43392 oasc.SyncStrategy.syncToMe https://127.0.0.1:43392/_/collection1/ has no replicas
   [junit4]   2> 447540 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43392/_/collection1/ shard1
   [junit4]   2> 447540 T2737 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 447540 T2706 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 447541 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447543 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447544 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447645 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447646 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447694 T2737 oasc.ZkController.register We are https://127.0.0.1:43392/_/collection1/ and leader is https://127.0.0.1:43392/_/collection1/
   [junit4]   2> 447694 T2737 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43392/_
   [junit4]   2> 447694 T2737 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 447694 T2737 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 447694 T2737 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447696 T2737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 447696 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 447697 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43392/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43392__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 447698 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447793 T2706 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 447794 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 447796 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 447797 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45364
   [junit4]   2> 447799 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/'
   [junit4]   2> 447799 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447799 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447809 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/solr.xml
   [junit4]   2> 447825 T2706 oasc.CoreContainer.<init> New CoreContainer 429451
   [junit4]   2> 447826 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/]
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 447828 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 447828 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 447829 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 447830 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 447830 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 447830 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 447830 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 447831 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447832 T2750 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e891c7c name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447832 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447834 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447834 T2752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c8afb5c name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447835 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447838 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448839 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45364__
   [junit4]   2> 448840 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45364__
   [junit4]   2> 448842 T2739 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448842 T2730 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448842 T2752 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448845 T2753 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 448845 T2753 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2753 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 448847 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45364/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45364__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 448848 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448949 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 448949 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 448949 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 449846 T2753 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 449847 T2753 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 449847 T2753 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 449847 T2753 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 449848 T2753 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 449848 T2753 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/'
   [junit4]   2> 449849 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 449849 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 449850 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 449869 T2753 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 449878 T2753 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 449879 T2753 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 449883 T2753 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 449997 T2753 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 449997 T2753 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 449997 T2753 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 450001 T2753 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 450002 T2753 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 450016 T2753 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450017 T2753 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450019 T2753 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450024 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450024 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450024 T2753 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 450024 T2753 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 450025 T2753 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/
   [junit4]   2> 450025 T2753 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 450025 T2753 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1
   [junit4]   2> 450026 T2753 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index/
   [junit4]   2> 450026 T2753 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 450026 T2753 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index
   [junit4]   2> 450027 T2753 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 450037 T2753 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 450037 T2753 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450039 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 450041 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450041 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450041 T2753 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 450041 T2753 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 450043 T2753 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 450043 T2753 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 450044 T2753 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450046 T2753 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 450046 T2753 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 450047 T2753 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 450048 T2753 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 450048 T2753 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450048 T2753 oass.SolrIndexSearcher.<init> Opening Searcher@31e15422[collection1] main
   [junit4]   2> 450049 T2753 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 450049 T2753 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 450050 T2753 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450051 T2753 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 450051 T2753 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450051 T2753 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 450051 T2753 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 450053 T2754 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31e15422[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 450055 T2753 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 450055 T2757 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:45364/_ collection:collection1 shard:shard2
   [junit4]   2> 450056 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 450056 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450056 T2757 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 450059 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450073 T2757 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 450073 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450074 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 450076 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C895 name=collection1 org.apache.solr.core.SolrCore@4e296a86 url=https://127.0.0.1:45364/_/collection1 node=127.0.0.1:45364__ C895_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:45364/_, core=collection1, node_name=127.0.0.1:45364__}
   [junit4]   2> 450076 T2757 C895 P45364 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:45364/_/collection1/
   [junit4]   2> 450076 T2757 C895 P45364 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 450077 T2757 C895 P45364 oasc.SyncStrategy.syncToMe https://127.0.0.1:45364/_/collection1/ has no replicas
   [junit4]   2> 450077 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:45364/_/collection1/ shard2
   [junit4]   2> 450077 T2757 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 450081 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450084 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450185 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450185 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450185 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450232 T2757 oasc.ZkController.register We are https://127.0.0.1:45364/_/collection1/ and leader is https://127.0.0.1:45364/_/collection1/
   [junit4]   2> 450233 T2757 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:45364/_
   [junit4]   2> 450233 T2757 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 450233 T2757 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 450233 T2757 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2757 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450235 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450236 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45364/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45364__",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 450238 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450328 T2706 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 450328 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 450331 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 450332 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38817
   [junit4]   2> 450334 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 450334 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 450334 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004
   [junit4]   2> 450335 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/'
   [junit4]   2> 450339 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450339 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450339 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450346 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/solr.xml
   [junit4]   2> 450354 T2706 oasc.CoreContainer.<init> New CoreContainer 1129178407
   [junit4]   2> 450354 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/]
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 450358 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 450358 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 450359 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 450359 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 450359 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 450360 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 450361 T2768 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@770558ae name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 450361 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 450362 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 450363 T2770 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26e0013d name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 450363 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 450366 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451368 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38817__
   [junit4]   2> 451369 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38817__
   [junit4]   2> 451371 T2739 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2730 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2752 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2770 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451375 T2771 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 451375 T2771 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2771 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451376 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38817/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38817__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 451378 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451479 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452376 T2771 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 452376 T2771 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 452377 T2771 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452377 T2771 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 452378 T2771 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452378 T2771 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/'
   [junit4]   2> 452378 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 452379 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 452379 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 452407 T2771 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 452416 T2771 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 452418 T2771 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 452422 T2771 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 452540 T2771 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452540 T2771 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452541 T2771 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 452544 T2771 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 452546 T2771 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 452553 T2771 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452555 T2771 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452558 T2771 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452558 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452559 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452559 T2771 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452560 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452560 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452560 T2771 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 452560 T2771 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 452561 T2771 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/
   [junit4]   2> 452561 T2771 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 452561 T2771 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2
   [junit4]   2> 452562 T2771 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index/
   [junit4]   2> 452562 T2771 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 452563 T2771 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index
   [junit4]   2> 452563 T2771 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 452571 T2771 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 452571 T2771 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452574 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452577 T2771 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 452577 T2771 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 452582 T2771 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 452583 T2771 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 452583 T2771 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 452584 T2771 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 452585 T2771 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 452585 T2771 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452586 T2771 oass.SolrIndexSearcher.<init> Opening Searcher@40e852a5[collection1] main
   [junit4]   2> 452586 T2771 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 452587 T2771 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452587 T2771 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 452587 T2771 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 452588 T2771 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 452588 T2771 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452588 T2771 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 452589 T2771 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452589 T2771 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 452589 T2771 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 452591 T2772 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40e852a5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 452593 T2771 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 452594 T2775 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38817/_ collection:collection1 shard:shard1
   [junit4]   2> 452594 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 452594 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 452595 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 452598 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 452598 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2775 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 452599 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 452599 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C896 name=collection1 org.apache.solr.core.SolrCore@3ee53a83 url=https://127.0.0.1:38817/_/collection1 node=127.0.0.1:38817__ C896_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:38817/_, core=collection1, node_name=127.0.0.1:38817__}
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38817/_/collection1/
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.syncToMe https://127.0.0.1:38817/_/collection1/ has no replicas
   [junit4]   2> 452600 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38817/_/collection1/ shard1
   [junit4]   2> 452600 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 452601 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452603 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452604 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452705 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452706 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452706 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452705 T2770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452753 T2775 oasc.ZkController.register We are https://127.0.0.1:38817/_/collection1/ and leader is https://127.0.0.1:38817/_/collection1/
   [junit4]   2> 452754 T2775 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38817/_
   [junit4]   2> 452754 T2775 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 452754 T2775 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 452754 T2775 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 452756 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 452756 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38817/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38817__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 452757 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452858 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452859 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452859 T2730 oascc.ZkStateReader$2.process A 

[...truncated too long message...]

jetty9
   [junit4]   2> 739816 T3579 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 739817 T3579 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 740063 T3654 C2657 P42194 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 740064 T3654 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 740064 T3654 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@63da0c51
   [junit4]   2> 740065 T3654 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=79,cumulative_deletesById=52,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=7948,transaction_logs_total_number=2}
   [junit4]   2> 740066 T3654 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 740066 T3654 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 740067 T3654 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 740068 T3654 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 740069 T3654 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 740069 T3654 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12;done=false>>]
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674;done=false>>]
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674
   [junit4]   2> 740071 T3654 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 740071 T3654 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene49), sim=DefaultSimilarity, locale=sr_ME, timezone=Africa/Nouakchott
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=66964352,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestReloadAndDeleteDocs, TestIBSimilarityFactory, DocExpirationUpdateProcessorFactoryTest, TestExtendedDismaxParser, DistributedDebugComponentTest, TestFunctionQuery, TestRandomFaceting, TestPostingsSolrHighlighter, TestCoreContainer, SolrRequestParserTest, TestSuggestSpellingConverter, OverseerStatusTest, NumericFieldsTest, MultiTermTest, TestBadConfig, DistribCursorPagingTest, TestXIncludeConfig, HdfsWriteToMultipleCollectionsTest, TestClassNameShortening, TestDistributedGrouping, TestLRUCache, CurrencyFieldXmlFileTest, DOMUtilTest, DeleteReplicaTest, TestOverriddenPrefixQueryForCustomFieldType, InfoHandlerTest, DirectSolrConnectionTest, TestFreeTextSuggestions, LegacyHTMLStripCharFilterTest, TestDynamicFieldResource, TestQuerySenderNoQuery, QueryParsingTest, EnumFieldTest, TestSweetSpotSimilarityFactory, TestSolrQueryParserDefaultOperatorResource, RequestHandlersTest, DeleteInactiveReplicaTest, TestStandardQParsers, DistributedTermsComponentTest, SchemaVersionSpecificBehaviorTest, TestRecovery, TestLazyCores, AutoCommitTest, MigrateRouteKeyTest, PreAnalyzedFieldTest, CopyFieldTest, SolrInfoMBeanTest, ResponseHeaderTest, LoggingHandlerTest, SolrIndexConfigTest, TestExpandComponent, TestIndexSearcher, TestRandomMergePolicy, TestReversedWildcardFilterFactory, TestNonNRTOpen, AliasIntegrationTest, SolrCoreTest, DistributedSuggestComponentTest, TestExceedMaxTermLength, BasicDistributedZk2Test, TestGroupingSearch, TestRealTimeGet, PolyFieldTest, RankQueryTest, TestRTGBase, CacheHeaderTest, DistributedQueryElevationComponentTest, TestSolrXmlPersistence, TestTolerantSearch, TestFieldTypeResource, TestMergePolicyConfig, TestNonDefinedSimilarityFactory, TestCollationField, HardAutoCommitTest, TestSolrQueryParserResource, SimplePostToolTest, SampleTest, TestDocBasedVersionConstraints, TestCSVLoader, IndexSchemaRuntimeFieldTest, TestOmitPositions, TestWriterPerf, TestFiltering, TestZkChroot, TestManagedResource, SearchHandlerTest, TestPseudoReturnFields, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, FileBasedSpellCheckerTest, TestMissingGroups, BlockCacheTest, TestIndexingPerformance, DistributedQueryComponentCustomSortTest, TestLFUCache, ConvertedLegacyTest, TestBinaryField, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 89.70s, 1 test, 1 failure <<< FAILURES!

[...truncated 697 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 2 failures, 37 ignored (16 assumptions)

Total time: 70 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_05) - Build # 10683 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10683/
Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"state":"active","base_url":"http://127.0.0.1:38061","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:38061_","leader":"true"}]; clusterState: live nodes:[127.0.0.1:40801_, 127.0.0.1:38061_] collections:{control_collection=DocCollection(control_collection)={   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:38061",           "core":"collection1",           "node_name":"127.0.0.1:38061_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{"core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:40801",           "core":"collection1",           "node_name":"127.0.0.1:40801_",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:51423",           "core":"collection1",           "node_name":"127.0.0.1:51423_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:38061",           "core":"c8n_1x2_shard1_replica1",           "node_name":"127.0.0.1:38061_"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:40801",           "core":"c8n_1x2_shard1_replica2",           "node_name":"127.0.0.1:40801_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:40801",           "core":"c8n_1x3_shard1_replica1",           "node_name":"127.0.0.1:40801_",           "leader":"true"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:51423",           "core":"c8n_1x3_shard1_replica2",           "node_name":"127.0.0.1:51423_"},         "core_node3":{           "state":"active",           "base_url":"http://127.0.0.1:38061",           "core":"c8n_1x3_shard1_replica3",           "node_name":"127.0.0.1:38061_"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:51423",           "core":"c8n_1x3_lf_shard1_replica2",           "node_name":"127.0.0.1:51423_"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:40801",           "core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:40801_"},         "core_node3":{           "state":"active",           "base_url":"http://127.0.0.1:38061",           "core":"c8n_1x3_lf_shard1_replica3",           "node_name":"127.0.0.1:38061_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"state":"active","base_url":"http://127.0.0.1:38061","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:38061_","leader":"true"}]; clusterState: live nodes:[127.0.0.1:40801_, 127.0.0.1:38061_] collections:{control_collection=DocCollection(control_collection)={
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:38061",
          "core":"collection1",
          "node_name":"127.0.0.1:38061_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{"core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:40801",
          "core":"collection1",
          "node_name":"127.0.0.1:40801_",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:51423",
          "core":"collection1",
          "node_name":"127.0.0.1:51423_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:38061",
          "core":"c8n_1x2_shard1_replica1",
          "node_name":"127.0.0.1:38061_"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:40801",
          "core":"c8n_1x2_shard1_replica2",
          "node_name":"127.0.0.1:40801_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:40801",
          "core":"c8n_1x3_shard1_replica1",
          "node_name":"127.0.0.1:40801_",
          "leader":"true"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:51423",
          "core":"c8n_1x3_shard1_replica2",
          "node_name":"127.0.0.1:51423_"},
        "core_node3":{
          "state":"active",
          "base_url":"http://127.0.0.1:38061",
          "core":"c8n_1x3_shard1_replica3",
          "node_name":"127.0.0.1:38061_"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:51423",
          "core":"c8n_1x3_lf_shard1_replica2",
          "node_name":"127.0.0.1:51423_"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:40801",
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:40801_"},
        "core_node3":{
          "state":"active",
          "base_url":"http://127.0.0.1:38061",
          "core":"c8n_1x3_lf_shard1_replica3",
          "node_name":"127.0.0.1:38061_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}}
	at __randomizedtesting.SeedInfo.seed([B699B0775A4A31C6:377F3E6F2D1551FA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:368)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11458 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/init-core-data-001
   [junit4]   2> 516337 T2344 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 516339 T2344 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 516339 T2344 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 516340 T2345 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 516440 T2344 oasc.ZkTestServer.run start zk server on port:60278
   [junit4]   2> 516441 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516444 T2351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a130336 name:ZooKeeperConnection Watcher:127.0.0.1:60278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516444 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516445 T2344 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 516448 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516449 T2353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b341c5e name:ZooKeeperConnection Watcher:127.0.0.1:60278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516449 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516450 T2344 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 516452 T2344 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 516454 T2344 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 516455 T2344 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 516456 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 516456 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 516458 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 516458 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 516460 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 516460 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 516461 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 516461 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 516462 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 516462 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 516463 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 516464 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 516465 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 516465 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 516466 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 516467 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 516468 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 516468 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 516469 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 516470 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 516471 T2344 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 516471 T2344 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 516716 T2344 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 516720 T2344 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38637
   [junit4]   2> 516721 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 516721 T2344 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 516721 T2344 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002
   [junit4]   2> 516721 T2344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/'
   [junit4]   2> 516741 T2344 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/solr.xml
   [junit4]   2> 516747 T2344 oasc.CoreContainer.<init> New CoreContainer 1863500737
   [junit4]   2> 516748 T2344 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/]
   [junit4]   2> 516748 T2344 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 516748 T2344 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 516749 T2344 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 516749 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 516749 T2344 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 516749 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 516749 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 516750 T2344 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 516750 T2344 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 516750 T2344 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 516751 T2344 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 516751 T2344 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 516752 T2344 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60278/solr
   [junit4]   2> 516752 T2344 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 516753 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516754 T2364 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74ad16d9 name:ZooKeeperConnection Watcher:127.0.0.1:60278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516754 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516756 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516757 T2366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3105d609 name:ZooKeeperConnection Watcher:127.0.0.1:60278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516757 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516758 T2344 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 516760 T2344 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 516764 T2344 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 516766 T2344 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 516768 T2344 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 516769 T2344 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 516770 T2344 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38061_
   [junit4]   2> 516771 T2344 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38061_
   [junit4]   2> 516772 T2344 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 516773 T2344 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 516774 T2344 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38061_
   [junit4]   2> 516775 T2344 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 516776 T2344 oasc.Overseer.start Overseer (id=92093715750518787-127.0.0.1:38061_-n_0000000000) starting
   [junit4]   2> 516777 T2344 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 516781 T2368 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 516781 T2344 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 516782 T2344 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 516783 T2344 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 516785 T2367 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 516786 T2369 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 516786 T2369 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 516787 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 516787 T2369 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 516788 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 516788 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38061",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38061_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 516788 T2367 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 516788 T2367 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 516790 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 516790 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 517787 T2369 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 517788 T2369 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 517788 T2369 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 517788 T2369 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 517789 T2369 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 517789 T2369 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/collection1/'
   [junit4]   2> 517790 T2369 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 517790 T2369 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 517790 T2369 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 517810 T2369 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 517820 T2369 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 517822 T2369 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 517826 T2369 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 517917 T2369 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 517917 T2369 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 517918 T2369 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 517921 T2369 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 517922 T2369 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 517929 T2369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 517931 T2369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 517933 T2369 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 517933 T2369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 517934 T2369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 517934 T2369 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 517934 T2369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 517935 T2369 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 517935 T2369 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 517935 T2369 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 517935 T2369 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/control/data/
   [junit4]   2> 517935 T2369 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@256b1693
   [junit4]   2> 517936 T2369 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/control/data
   [junit4]   2> 517936 T2369 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/control/data/index/
   [junit4]   2> 517936 T2369 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 517936 T2369 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/control/data/index
   [junit4]   2> 517937 T2369 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=10.5634765625, floorSegmentMB=0.25, forceMergeDeletesPctAllowed=10.241681063914715, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 517937 T2369 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@705d4b1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@350b06c1),segFN=segments_1,generation=1}
   [junit4]   2> 517937 T2369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 517939 T2369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 517939 T2369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 517939 T2369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 517940 T2369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 517940 T2369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 517940 T2369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 517940 T2369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 517940 T2369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 517940 T2369 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 517941 T2369 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 517941 T2369 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 517941 T2369 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 517941 T2369 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 517942 T2369 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 517942 T2369 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 517942 T2369 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 517944 T2369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 517945 T2369 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 517945 T2369 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 517945 T2369 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=40.46484375, floorSegmentMB=1.8017578125, forceMergeDeletesPctAllowed=7.796274678988942, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 517946 T2369 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@705d4b1a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@350b06c1),segFN=segments_1,generation=1}
   [junit4]   2> 517946 T2369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 517946 T2369 oass.SolrIndexSearcher.<init> Opening Searcher@7cd152f0[collection1] main
   [junit4]   2> 517946 T2369 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 517947 T2369 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 517947 T2369 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 517947 T2369 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 517948 T2369 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 517948 T2369 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 517948 T2369 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 517948 T2369 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 517949 T2369 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 517949 T2369 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 517950 T2370 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7cd152f0[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 517950 T2369 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 517951 T2373 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38061 collection:control_collection shard:shard1
   [junit4]   2> 517951 T2344 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 517952 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 517952 T2373 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 517958 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 517959 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 517959 T2376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ad8bd3 name:ZooKeeperConnection Watcher:127.0.0.1:60278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517960 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 517960 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 517960 T2344 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 517960 T2373 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 517961 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C962 name=collection1 org.apache.solr.core.SolrCore@7a30da00 url=http://127.0.0.1:38061/collection1 node=127.0.0.1:38061_ C962_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38061, core=collection1, node_name=127.0.0.1:38061_}
   [junit4]   2> 517961 T2373 C962 P38061 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38061/collection1/
   [junit4]   2> 517961 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 517961 T2373 C962 P38061 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 517961 T2373 C962 P38061 oasc.SyncStrategy.syncToMe http://127.0.0.1:38061/collection1/ has no replicas
   [junit4]   2> 517961 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38061/collection1/ shard1
   [junit4]   2> 517961 T2344 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 517961 T2373 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 517962 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 517964 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 517965 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 518067 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518067 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518115 T2373 oasc.ZkController.register We are http://127.0.0.1:38061/collection1/ and leader is http://127.0.0.1:38061/collection1/
   [junit4]   2> 518115 T2373 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38061
   [junit4]   2> 518115 T2373 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 518116 T2373 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 518116 T2373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 518117 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 518117 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 518117 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 518118 T2373 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 518119 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 518119 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38061",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38061_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 518121 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 518211 T2344 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 518213 T2344 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 518217 T2344 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44347
   [junit4]   2> 518218 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 518218 T2344 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 518218 T2344 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003
   [junit4]   2> 518219 T2344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/'
   [junit4]   2> 518222 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518222 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518231 T2344 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/solr.xml
   [junit4]   2> 518237 T2344 oasc.CoreContainer.<init> New CoreContainer 1018591868
   [junit4]   2> 518237 T2344 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/]
   [junit4]   2> 518238 T2344 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 518238 T2344 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 518238 T2344 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 518238 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 518238 T2344 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 518239 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 518239 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 518239 T2344 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 518239 T2344 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 518240 T2344 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 518240 T2344 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 518240 T2344 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 518241 T2344 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60278/solr
   [junit4]   2> 518241 T2344 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 518242 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 518243 T2387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43c429c8 name:ZooKeeperConnection Watcher:127.0.0.1:60278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 518243 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 518244 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 518245 T2389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@119300cf name:ZooKeeperConnection Watcher:127.0.0.1:60278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 518245 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 518248 T2344 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 519250 T2344 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51423_
   [junit4]   2> 519251 T2344 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51423_
   [junit4]   2> 519253 T2366 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 519253 T2389 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 519253 T2376 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 519257 T2390 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 519257 T2390 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 519258 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 519258 T2390 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 519258 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 519259 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 519259 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 519260 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51423",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51423_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 519260 T2367 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 519260 T2367 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 519261 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 519363 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 519363 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 519363 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520259 T2390 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 520259 T2390 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 520260 T2390 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 520260 T2390 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 520260 T2390 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 520260 T2390 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/collection1/'
   [junit4]   2> 520261 T2390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 520261 T2390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 520261 T2390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 520289 T2390 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 520300 T2390 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 520301 T2390 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 520305 T2390 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 520439 T2390 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 520440 T2390 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 520440 T2390 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 520446 T2390 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 520448 T2390 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 520458 T2390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 520461 T2390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 520464 T2390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 520465 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 520465 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 520465 T2390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 520466 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 520466 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 520466 T2390 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 520467 T2390 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 520467 T2390 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty1/
   [junit4]   2> 520467 T2390 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@256b1693
   [junit4]   2> 520468 T2390 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty1
   [junit4]   2> 520468 T2390 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty1/index/
   [junit4]   2> 520468 T2390 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 520469 T2390 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty1/index
   [junit4]   2> 520469 T2390 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=10.5634765625, floorSegmentMB=0.25, forceMergeDeletesPctAllowed=10.241681063914715, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 520470 T2390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@63e9943c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f42e9fb),segFN=segments_1,generation=1}
   [junit4]   2> 520470 T2390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 520473 T2390 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 520474 T2390 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 520474 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 520474 T2390 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 520474 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 520475 T2390 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 520475 T2390 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 520475 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 520476 T2390 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 520476 T2390 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 520477 T2390 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 520477 T2390 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 520477 T2390 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 520477 T2390 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 520481 T2390 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 520481 T2390 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 520484 T2390 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 520485 T2390 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 520486 T2390 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 520486 T2390 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=40.46484375, floorSegmentMB=1.8017578125, forceMergeDeletesPctAllowed=7.796274678988942, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 520487 T2390 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@63e9943c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f42e9fb),segFN=segments_1,generation=1}
   [junit4]   2> 520487 T2390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 520487 T2390 oass.SolrIndexSearcher.<init> Opening Searcher@436bba0c[collection1] main
   [junit4]   2> 520488 T2390 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 520488 T2390 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 520489 T2390 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 520489 T2390 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 520489 T2390 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 520490 T2390 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 520490 T2390 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 520490 T2390 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 520490 T2390 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 520491 T2390 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 520493 T2391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@436bba0c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 520495 T2390 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 520496 T2394 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51423 collection:collection1 shard:shard2
   [junit4]   2> 520497 T2344 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 520497 T2394 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 520497 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 520500 T2394 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 520502 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520502 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520502 T2394 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 520502 T2394 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 520502 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C963 name=collection1 org.apache.solr.core.SolrCore@dc695bb url=http://127.0.0.1:51423/collection1 node=127.0.0.1:51423_ C963_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51423, core=collection1, node_name=127.0.0.1:51423_}
   [junit4]   2> 520503 T2394 C963 P51423 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51423/collection1/
   [junit4]   2> 520503 T2394 C963 P51423 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 520503 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 520503 T2394 C963 P51423 oasc.SyncStrategy.syncToMe http://127.0.0.1:51423/collection1/ has no replicas
   [junit4]   2> 520504 T2394 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51423/collection1/ shard2
   [junit4]   2> 520504 T2394 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 520505 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520508 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520510 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520611 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520611 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520611 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520659 T2394 oasc.ZkController.register We are http://127.0.0.1:51423/collection1/ and leader is http://127.0.0.1:51423/collection1/
   [junit4]   2> 520659 T2394 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51423
   [junit4]   2> 520659 T2394 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 520659 T2394 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 520660 T2394 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 520661 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520661 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520661 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520661 T2394 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 520662 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 520662 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51423",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51423_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 520664 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 520765 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520765 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520765 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520769 T2344 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 520770 T2344 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 520771 T2344 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41897
   [junit4]   2> 520772 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 520772 T2344 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 520772 T2344 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004
   [junit4]   2> 520773 T2344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/'
   [junit4]   2> 520785 T2344 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/solr.xml
   [junit4]   2> 520792 T2344 oasc.CoreContainer.<init> New CoreContainer 64614375
   [junit4]   2> 520792 T2344 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/]
   [junit4]   2> 520793 T2344 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 520793 T2344 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 520793 T2344 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 520793 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 520794 T2344 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 520794 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 520794 T2344 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 520794 T2344 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 520795 T2344 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 520795 T2344 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 520795 T2344 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 520796 T2344 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 520796 T2344 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60278/solr
   [junit4]   2> 520796 T2344 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 520797 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 520798 T2406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5678b624 name:ZooKeeperConnection Watcher:127.0.0.1:60278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 520798 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 520800 T2344 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 520800 T2408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64b1f614 name:ZooKeeperConnection Watcher:127.0.0.1:60278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 520801 T2344 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 520805 T2344 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 521808 T2344 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40801_
   [junit4]   2> 521809 T2344 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40801_
   [junit4]   2> 521813 T2376 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 521814 T2389 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 521814 T2408 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 521813 T2366 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 521821 T2409 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 521821 T2409 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 521823 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521823 T2409 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 521823 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521824 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521826 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 521826 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40801",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40801_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 521826 T2367 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 521827 T2367 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 521829 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 521931 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 521931 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 521931 T2408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 521931 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 522824 T2409 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 522824 T2409 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 522825 T2409 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 522825 T2409 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 522827 T2409 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 522827 T2409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/collection1/'
   [junit4]   2> 522828 T2409 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 522828 T2409 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 522829 T2409 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 522855 T2409 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 522863 T2409 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 522864 T2409 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 522867 T2409 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 522961 T2409 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 522961 T2409 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 522961 T2409 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 522965 T2409 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 522966 T2409 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 522972 T2409 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 522974 T2409 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 522975 T2409 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 522976 T2409 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 522976 T2409 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 522976 T2409 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 522977 T2409 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 522977 T2409 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 522977 T2409 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 522977 T2409 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 522977 T2409 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty2/
   [junit4]   2> 522977 T2409 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@256b1693
   [junit4]   2> 522978 T2409 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty2
   [junit4]   2> 522978 T2409 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty2/index/
   [junit4]   2> 522978 T2409 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 522979 T2409 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001/tempDir-001/jetty2/index
   [junit4]   2> 522979 T2409 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=10.5634765625, floorSegmentMB=0.25, forceMergeDeletesPctAllowed=10.241681063914715, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 522979 T2409 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@23b5bf85 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@266ab876),segFN=segments_1,generation=1}
   [junit4]   2> 522980 T2409 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 522982 T2409 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 522982 T2409 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 522982 T2409 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 522982 T2409 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 522982 T2409 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 522983 T2409 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 522983 T2409 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 522983 T2409 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 522983 T2409 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 522983 T2409 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 522984 T2409 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 522984 T2409 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 522984 T2409 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 522984 T2409 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 522984 T2409 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 522985 T2409 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 522986 T2409 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 522987 T2409 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 522987 T2409 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 522988 T2409 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=40.46484375, floorSegmentMB=1.8017578125, forceMergeDeletesPctAllowed=7.796274678988942, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 522988 T2409 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@23b5bf85 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@266ab876),segFN=segments_1,generation=1}
   [junit4]   2> 522988 T2409 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 522988 T2409 oass.SolrIndexSearcher.<init> Opening Searcher@740ab36f[collection1] main
   [junit4]   2> 522989 T2409 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 522989 T2409 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 522990 T2409 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 522990 T2409 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 522990 T2409 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 522990 T2409 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 522991 T2409 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 522991 T2409 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 522991 T2409 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 522991 T2409 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 522993 T2410 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@740ab36f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 522994 T2409 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 522994 T2413 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40801 collection:collection1 shard:shard1
   [junit4]   2> 522995 T2344 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 522995 T2344 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 522995 T2413 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 522996 T2344 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 522997 T2344 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 522997 T2344 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 522998 T2344 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 522998 T2413 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 522999 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 522999 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523000 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523000 T2413 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 523000 T2413 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C964 name=collection1 org.apache.solr.core.SolrCore@591aefea url=http://127.0.0.1:40801/collection1 node=127.0.0.1:40801_ C964_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40801, core=collection1, node_name=127.0.0.1:40801_}
   [junit4]   2> 523000 T2413 C964 P40801 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40801/collection1/
   [junit4]   2> 523000 T2413 C964 P40801 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 523001 T2413 C964 P40801 oasc.SyncStrategy.syncToMe http://127.0.0.1:40801/collection1/ has no replicas
   [junit4]   2> 523001 T2413 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40801/collection1/ shard1
   [junit4]   2> 523001 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 523001 T2413 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 523003 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523004 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523005 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523107 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 523107 T2366 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 523107 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 523107 T2408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 523155 T2413 oasc.ZkController.register We are http://127.0.0.1:40801/collection1/ and leader is http://127.0.0.1:40801/collection1/
   [junit4]   2> 523155 T2413 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40801
   [junit4]   2> 523155 T2413 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 523155 T2413 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 523155 T2413 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 523156 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523157 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523157 T2413 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 523157 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523158 T2367 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 523159 T2367 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40801",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40801_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 523163 T2366 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 523264 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes

[...truncated too long message...]

name":"127.0.0.1:40801_",
   [junit4]    >           "leader":"true"}}},
   [junit4]    >     "shard2":{
   [junit4]    >       "range":"0-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{"core_node1":{
   [junit4]    >           "state":"down",
   [junit4]    >           "base_url":"http://127.0.0.1:51423",
   [junit4]    >           "core":"collection1",
   [junit4]    >           "node_name":"127.0.0.1:51423_",
   [junit4]    >           "leader":"true"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "autoCreated":"true",
   [junit4]    >   "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={
   [junit4]    >   "replicationFactor":"2",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "state":"active",
   [junit4]    >           "base_url":"http://127.0.0.1:38061",
   [junit4]    >           "core":"c8n_1x2_shard1_replica1",
   [junit4]    >           "node_name":"127.0.0.1:38061_"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "state":"active",
   [junit4]    >           "base_url":"http://127.0.0.1:40801",
   [junit4]    >           "core":"c8n_1x2_shard1_replica2",
   [junit4]    >           "node_name":"127.0.0.1:40801_",
   [junit4]    >           "leader":"true"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={
   [junit4]    >   "replicationFactor":"3",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "state":"active",
   [junit4]    >           "base_url":"http://127.0.0.1:40801",
   [junit4]    >           "core":"c8n_1x3_shard1_replica1",
   [junit4]    >           "node_name":"127.0.0.1:40801_",
   [junit4]    >           "leader":"true"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "state":"down",
   [junit4]    >           "base_url":"http://127.0.0.1:51423",
   [junit4]    >           "core":"c8n_1x3_shard1_replica2",
   [junit4]    >           "node_name":"127.0.0.1:51423_"},
   [junit4]    >         "core_node3":{
   [junit4]    >           "state":"active",
   [junit4]    >           "base_url":"http://127.0.0.1:38061",
   [junit4]    >           "core":"c8n_1x3_shard1_replica3",
   [junit4]    >           "node_name":"127.0.0.1:38061_"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={
   [junit4]    >   "replicationFactor":"3",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "state":"down",
   [junit4]    >           "base_url":"http://127.0.0.1:51423",
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica2",
   [junit4]    >           "node_name":"127.0.0.1:51423_"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "state":"down",
   [junit4]    >           "base_url":"http://127.0.0.1:40801",
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica1",
   [junit4]    >           "node_name":"127.0.0.1:40801_"},
   [junit4]    >         "core_node3":{
   [junit4]    >           "state":"active",
   [junit4]    >           "base_url":"http://127.0.0.1:38061",
   [junit4]    >           "core":"c8n_1x3_lf_shard1_replica3",
   [junit4]    >           "node_name":"127.0.0.1:38061_",
   [junit4]    >           "leader":"true"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1"}}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B699B0775A4A31C6:377F3E6F2D1551FA]:0)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:368)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 623229 T2344 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-B699B0775A4A31C6-001
   [junit4]   2> 106896 T2343 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 623885 T2656 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:316)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
   [junit4]   2> 
   [junit4]   2> 623885 T2408 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene49), sim=DefaultSimilarity, locale=fr_FR, timezone=Africa/Tunis
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=149749040,total=247513088
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues, TestSurroundQueryParser, DocumentBuilderTest, RemoteQueryErrorTest, TestSearcherReuse, TestReRankQParserPlugin, PrimUtilsTest, SignatureUpdateProcessorFactoryTest, UUIDFieldTest, UnloadDistributedZkTest, AssignTest, StressHdfsTest, TestArbitraryIndexDir, BadIndexSchemaTest, TestDistributedSearch, TestAnalyzedSuggestions, TestComponentsName, TestStressVersions, TestSolrIndexConfig, ClusterStateUpdateTest, TermsComponentTest, JSONWriterTest, MoreLikeThisHandlerTest, BadCopyFieldTest, TestDistribDocBasedVersion, TestSolrXml, TestDefaultSimilarityFactory, BlockDirectoryTest, StandardRequestHandlerTest, TestHighlightDedupGrouping, TestDistributedMissingSort, HdfsCollectionsAPIDistributedZkTest, PeerSyncTest, TestFieldTypeCollectionResource, TestWordDelimiterFilterFactory, TestFuzzyAnalyzedSuggestions, FastVectorHighlighterTest, TestBM25SimilarityFactory, CoreAdminHandlerTest, MBeansHandlerTest, TestRangeQuery, TestReload, IndexBasedSpellCheckerTest, ResourceLoaderTest, TestAddFieldRealTimeGet, RAMDirectoryFactoryTest, TestShardHandlerFactory, TestChildDocTransformer, TestBlendedInfixSuggestions, TestSearchPerf, SolrCoreCheckLockOnStartupTest, TestLMJelinekMercerSimilarityFactory, TestSort, TestSolrXMLSerializer, TestDynamicFieldCollectionResource, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, SpellCheckComponentTest, DateFieldTest, TestFieldCollectionResource, TestConfig, MinimalSchemaTest, ShardSplitTest, TestImplicitCoreProperties, RegexBoostProcessorTest, DisMaxRequestHandlerTest, TestDocSet, EchoParamsTest, SSLMigrationTest, HdfsLockFactoryTest, DocumentAnalysisRequestHandlerTest, TestHighFrequencyDictionaryFactory, LeaderElectionTest, CachingDirectoryFactoryTest, TestMiniSolrCloudCluster, TestManagedSchemaFieldResource, TestCopyFieldCollectionResource, CoreAdminCreateDiscoverTest, ZkCLITest, UpdateParamsTest, PluginInfoTest, TestFaceting, TestManagedStopFilterFactory, CoreMergeIndexesAdminHandlerTest, PrimitiveFieldTypeTest, HttpPartitionTest]
   [junit4] Completed on J0 in 107.66s, 1 test, 1 failure <<< FAILURES!

[...truncated 702 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 1 failure, 37 ignored (16 assumptions)

Total time: 67 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60) - Build # 10682 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10682/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"state":"active","base_url":"http://127.0.0.1:35697/wvvc/j","core":"c8n_1x3_lf_shard1_replica2","node_name":"127.0.0.1:35697_wvvc%2Fj","leader":"true"}]; clusterState: live nodes:[127.0.0.1:35697_wvvc%2Fj, 127.0.0.1:58454_wvvc%2Fj] collections:{control_collection=DocCollection(control_collection)={   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:47839/wvvc/j",           "core":"collection1",           "node_name":"127.0.0.1:47839_wvvc%2Fj",           "leader":"true"}}}},   "replicationFactor":"1",   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{"core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:58454/wvvc/j",           "core":"collection1",           "node_name":"127.0.0.1:58454_wvvc%2Fj",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:35697/wvvc/j",           "core":"collection1",           "node_name":"127.0.0.1:35697_wvvc%2Fj",           "leader":"true"}}}},   "replicationFactor":"1",   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:47839/wvvc/j",           "core":"c8n_1x2_shard1_replica2",           "node_name":"127.0.0.1:47839_wvvc%2Fj"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:35697/wvvc/j",           "core":"c8n_1x2_shard1_replica1",           "node_name":"127.0.0.1:35697_wvvc%2Fj",           "leader":"true"}}}},   "replicationFactor":"2",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:47839/wvvc/j",           "core":"c8n_1x3_shard1_replica1",           "node_name":"127.0.0.1:47839_wvvc%2Fj"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:35697/wvvc/j",           "core":"c8n_1x3_shard1_replica2",           "node_name":"127.0.0.1:35697_wvvc%2Fj",           "leader":"true"},         "core_node3":{           "state":"active",           "base_url":"http://127.0.0.1:58454/wvvc/j",           "core":"c8n_1x3_shard1_replica3",           "node_name":"127.0.0.1:58454_wvvc%2Fj"}}}},   "replicationFactor":"3",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:58454/wvvc/j",           "core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:58454_wvvc%2Fj"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:35697/wvvc/j",           "core":"c8n_1x3_lf_shard1_replica2",           "node_name":"127.0.0.1:35697_wvvc%2Fj",           "leader":"true"},         "core_node3":{           "state":"down",           "base_url":"http://127.0.0.1:47839/wvvc/j",           "core":"c8n_1x3_lf_shard1_replica3",           "node_name":"127.0.0.1:47839_wvvc%2Fj"}}}},   "replicationFactor":"3",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"state":"active","base_url":"http://127.0.0.1:35697/wvvc/j","core":"c8n_1x3_lf_shard1_replica2","node_name":"127.0.0.1:35697_wvvc%2Fj","leader":"true"}]; clusterState: live nodes:[127.0.0.1:35697_wvvc%2Fj, 127.0.0.1:58454_wvvc%2Fj] collections:{control_collection=DocCollection(control_collection)={
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:47839/wvvc/j",
          "core":"collection1",
          "node_name":"127.0.0.1:47839_wvvc%2Fj",
          "leader":"true"}}}},
  "replicationFactor":"1",
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{"core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:58454/wvvc/j",
          "core":"collection1",
          "node_name":"127.0.0.1:58454_wvvc%2Fj",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:35697/wvvc/j",
          "core":"collection1",
          "node_name":"127.0.0.1:35697_wvvc%2Fj",
          "leader":"true"}}}},
  "replicationFactor":"1",
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:47839/wvvc/j",
          "core":"c8n_1x2_shard1_replica2",
          "node_name":"127.0.0.1:47839_wvvc%2Fj"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:35697/wvvc/j",
          "core":"c8n_1x2_shard1_replica1",
          "node_name":"127.0.0.1:35697_wvvc%2Fj",
          "leader":"true"}}}},
  "replicationFactor":"2",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:47839/wvvc/j",
          "core":"c8n_1x3_shard1_replica1",
          "node_name":"127.0.0.1:47839_wvvc%2Fj"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:35697/wvvc/j",
          "core":"c8n_1x3_shard1_replica2",
          "node_name":"127.0.0.1:35697_wvvc%2Fj",
          "leader":"true"},
        "core_node3":{
          "state":"active",
          "base_url":"http://127.0.0.1:58454/wvvc/j",
          "core":"c8n_1x3_shard1_replica3",
          "node_name":"127.0.0.1:58454_wvvc%2Fj"}}}},
  "replicationFactor":"3",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:58454/wvvc/j",
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:58454_wvvc%2Fj"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:35697/wvvc/j",
          "core":"c8n_1x3_lf_shard1_replica2",
          "node_name":"127.0.0.1:35697_wvvc%2Fj",
          "leader":"true"},
        "core_node3":{
          "state":"down",
          "base_url":"http://127.0.0.1:47839/wvvc/j",
          "core":"c8n_1x3_lf_shard1_replica3",
          "node_name":"127.0.0.1:47839_wvvc%2Fj"}}}},
  "replicationFactor":"3",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}}
	at __randomizedtesting.SeedInfo.seed([6A6ED270C4379229:EB885C68B368F215]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:368)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11183 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/init-core-data-001
   [junit4]   2> 451031 T2934 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /wvvc/j
   [junit4]   2> 451034 T2934 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 451035 T2934 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 451035 T2935 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 451135 T2934 oasc.ZkTestServer.run start zk server on port:52067
   [junit4]   2> 451136 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451138 T2941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f4bac7 name:ZooKeeperConnection Watcher:127.0.0.1:52067 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451138 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451139 T2934 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 451141 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451142 T2943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f82f5d name:ZooKeeperConnection Watcher:127.0.0.1:52067/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451142 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451142 T2934 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 451144 T2934 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 451146 T2934 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 451147 T2934 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 451148 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 451149 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 451151 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 451151 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 451153 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 451154 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 451155 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 451155 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 451157 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 451157 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 451159 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 451159 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 451161 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 451161 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 451163 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 451163 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 451165 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 451165 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 451167 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 451167 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 451168 T2934 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 451169 T2934 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 451430 T2934 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 451433 T2934 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46180
   [junit4]   2> 451434 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 451434 T2934 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 451434 T2934 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002
   [junit4]   2> 451435 T2934 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/'
   [junit4]   2> 451454 T2934 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/solr.xml
   [junit4]   2> 451483 T2934 oasc.CoreContainer.<init> New CoreContainer 11627604
   [junit4]   2> 451484 T2934 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/]
   [junit4]   2> 451485 T2934 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 451485 T2934 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 451485 T2934 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 451486 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 451486 T2934 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 451486 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 451486 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 451487 T2934 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 451487 T2934 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 451487 T2934 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 451488 T2934 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 451488 T2934 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 451488 T2934 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52067/solr
   [junit4]   2> 451489 T2934 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 451490 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451491 T2954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f2f5d name:ZooKeeperConnection Watcher:127.0.0.1:52067 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451491 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451493 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451494 T2956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161e28f name:ZooKeeperConnection Watcher:127.0.0.1:52067/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451494 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451495 T2934 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 451497 T2934 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 451499 T2934 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 451500 T2934 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 451502 T2934 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 451504 T2934 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 451505 T2934 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47839_wvvc%2Fj
   [junit4]   2> 451505 T2934 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47839_wvvc%2Fj
   [junit4]   2> 451507 T2934 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 451508 T2934 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 451510 T2934 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47839_wvvc%2Fj
   [junit4]   2> 451510 T2934 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 451511 T2934 oasc.Overseer.start Overseer (id=92093178313244675-127.0.0.1:47839_wvvc%2Fj-n_0000000000) starting
   [junit4]   2> 451513 T2934 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 451518 T2958 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 451519 T2934 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 451520 T2934 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 451521 T2934 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451522 T2957 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 451525 T2959 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 451525 T2959 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 451526 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451526 T2959 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451527 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451527 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47839/wvvc/j",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47839_wvvc%2Fj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451527 T2957 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 451527 T2957 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 451528 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451529 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 452526 T2959 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 452527 T2959 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 452527 T2959 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452527 T2959 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 452528 T2959 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452528 T2959 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/collection1/'
   [junit4]   2> 452529 T2959 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 452529 T2959 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 452530 T2959 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 452560 T2959 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 452588 T2959 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 452589 T2959 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 452597 T2959 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 452874 T2959 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452875 T2959 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452875 T2959 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 452880 T2959 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 452883 T2959 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 452893 T2959 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452896 T2959 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452899 T2959 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452900 T2959 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452900 T2959 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452900 T2959 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452901 T2959 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452901 T2959 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452901 T2959 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 452901 T2959 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 452901 T2959 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/control/data/
   [junit4]   2> 452902 T2959 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ef1d4c
   [junit4]   2> 452902 T2959 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/control/data
   [junit4]   2> 452902 T2959 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/control/data/index/
   [junit4]   2> 452903 T2959 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 452903 T2959 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/control/data/index
   [junit4]   2> 452903 T2959 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=14.2666015625, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=9.833457138872769, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 452904 T2959 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ec57d0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cdb38),segFN=segments_1,generation=1}
   [junit4]   2> 452904 T2959 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452906 T2959 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 452907 T2959 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 452907 T2959 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 452907 T2959 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452907 T2959 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452907 T2959 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 452908 T2959 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452908 T2959 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452908 T2959 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 452909 T2959 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 452909 T2959 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 452909 T2959 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 452910 T2959 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 452910 T2959 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 452911 T2959 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 452911 T2959 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 452914 T2959 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 452917 T2959 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 452917 T2959 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 452918 T2959 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 452918 T2959 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ec57d0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cdb38),segFN=segments_1,generation=1}
   [junit4]   2> 452918 T2959 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452919 T2959 oass.SolrIndexSearcher.<init> Opening Searcher@666fbb[collection1] main
   [junit4]   2> 452919 T2959 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 452920 T2959 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452920 T2959 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 452920 T2959 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 452921 T2959 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 452921 T2959 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452921 T2959 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 452922 T2959 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452922 T2959 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 452922 T2959 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 452924 T2960 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@666fbb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 452925 T2959 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 452926 T2963 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47839/wvvc/j collection:control_collection shard:shard1
   [junit4]   2> 452926 T2934 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 452927 T2963 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 452927 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 452930 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 452931 T2963 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 452931 T2966 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ab1bbe name:ZooKeeperConnection Watcher:127.0.0.1:52067/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 452931 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 452932 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452932 T2934 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 452932 T2963 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 452933 T2963 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C414 name=collection1 org.apache.solr.core.SolrCore@56e4a5 url=http://127.0.0.1:47839/wvvc/j/collection1 node=127.0.0.1:47839_wvvc%2Fj C414_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:47839/wvvc/j, core=collection1, node_name=127.0.0.1:47839_wvvc%2Fj}
   [junit4]   2> 452933 T2963 C414 P47839 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47839/wvvc/j/collection1/
   [junit4]   2> 452933 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 452933 T2963 C414 P47839 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 452934 T2963 C414 P47839 oasc.SyncStrategy.syncToMe http://127.0.0.1:47839/wvvc/j/collection1/ has no replicas
   [junit4]   2> 452934 T2934 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 452934 T2963 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47839/wvvc/j/collection1/ shard1
   [junit4]   2> 452934 T2963 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 452935 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452937 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452939 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453040 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 453040 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 453088 T2963 oasc.ZkController.register We are http://127.0.0.1:47839/wvvc/j/collection1/ and leader is http://127.0.0.1:47839/wvvc/j/collection1/
   [junit4]   2> 453088 T2963 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47839/wvvc/j
   [junit4]   2> 453089 T2963 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 453089 T2963 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 453089 T2963 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 453090 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453090 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453090 T2963 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 453091 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453092 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 453094 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47839/wvvc/j",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47839_wvvc%2Fj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 453096 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453198 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 453198 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 453212 T2934 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 453213 T2934 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 453215 T2934 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44588
   [junit4]   2> 453215 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 453216 T2934 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 453216 T2934 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003
   [junit4]   2> 453216 T2934 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/'
   [junit4]   2> 453229 T2934 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/solr.xml
   [junit4]   2> 453257 T2934 oasc.CoreContainer.<init> New CoreContainer 29026337
   [junit4]   2> 453258 T2934 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/]
   [junit4]   2> 453259 T2934 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 453259 T2934 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 453259 T2934 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 453260 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 453260 T2934 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 453260 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 453260 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 453261 T2934 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 453261 T2934 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 453262 T2934 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 453262 T2934 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 453262 T2934 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 453263 T2934 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52067/solr
   [junit4]   2> 453263 T2934 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 453264 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453265 T2977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ddec23 name:ZooKeeperConnection Watcher:127.0.0.1:52067 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453265 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453267 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453267 T2979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11311bf name:ZooKeeperConnection Watcher:127.0.0.1:52067/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453268 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453272 T2934 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 454274 T2934 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35697_wvvc%2Fj
   [junit4]   2> 454275 T2934 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35697_wvvc%2Fj
   [junit4]   2> 454278 T2979 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 454278 T2956 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 454278 T2966 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 454282 T2980 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 454283 T2980 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 454283 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 454283 T2980 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 454284 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 454284 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 454285 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 454286 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35697/wvvc/j",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35697_wvvc%2Fj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 454286 T2957 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 454286 T2957 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 454287 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 454389 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 454389 T2979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 454389 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455284 T2980 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 455284 T2980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 455285 T2980 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 455285 T2980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 455286 T2980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 455286 T2980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/collection1/'
   [junit4]   2> 455287 T2980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 455288 T2980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 455288 T2980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 455334 T2980 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 455365 T2980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 455366 T2980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 455376 T2980 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 455670 T2980 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 455671 T2980 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 455671 T2980 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455676 T2980 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 455678 T2980 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 455688 T2980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455691 T2980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455693 T2980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455694 T2980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455694 T2980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455695 T2980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455695 T2980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455696 T2980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455696 T2980 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 455696 T2980 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 455696 T2980 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty1/
   [junit4]   2> 455696 T2980 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ef1d4c
   [junit4]   2> 455697 T2980 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty1
   [junit4]   2> 455697 T2980 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty1/index/
   [junit4]   2> 455697 T2980 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 455698 T2980 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty1/index
   [junit4]   2> 455698 T2980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=14.2666015625, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=9.833457138872769, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 455698 T2980 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6489f8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@191e8b4),segFN=segments_1,generation=1}
   [junit4]   2> 455699 T2980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455701 T2980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 455702 T2980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 455702 T2980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 455702 T2980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455702 T2980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455702 T2980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 455703 T2980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455703 T2980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455703 T2980 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 455704 T2980 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 455704 T2980 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 455704 T2980 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 455705 T2980 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 455705 T2980 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 455705 T2980 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 455706 T2980 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 455708 T2980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 455710 T2980 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 455710 T2980 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 455711 T2980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 455712 T2980 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6489f8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@191e8b4),segFN=segments_1,generation=1}
   [junit4]   2> 455712 T2980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455712 T2980 oass.SolrIndexSearcher.<init> Opening Searcher@7f4293[collection1] main
   [junit4]   2> 455712 T2980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 455713 T2980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 455713 T2980 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 455714 T2980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 455714 T2980 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 455714 T2980 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 455715 T2980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 455715 T2980 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 455715 T2980 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 455715 T2980 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 455719 T2981 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f4293[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 455721 T2980 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 455721 T2984 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35697/wvvc/j collection:collection1 shard:shard2
   [junit4]   2> 455722 T2934 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 455722 T2984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 455722 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 455725 T2984 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 455726 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455727 T2984 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 455727 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455727 T2984 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 455727 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C415 name=collection1 org.apache.solr.core.SolrCore@10b7f3a url=http://127.0.0.1:35697/wvvc/j/collection1 node=127.0.0.1:35697_wvvc%2Fj C415_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35697/wvvc/j, core=collection1, node_name=127.0.0.1:35697_wvvc%2Fj}
   [junit4]   2> 455727 T2984 C415 P35697 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35697/wvvc/j/collection1/
   [junit4]   2> 455728 T2984 C415 P35697 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 455728 T2984 C415 P35697 oasc.SyncStrategy.syncToMe http://127.0.0.1:35697/wvvc/j/collection1/ has no replicas
   [junit4]   2> 455728 T2984 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35697/wvvc/j/collection1/ shard2
   [junit4]   2> 455728 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 455728 T2984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 455730 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455731 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455733 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455835 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455835 T2979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455835 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455882 T2984 oasc.ZkController.register We are http://127.0.0.1:35697/wvvc/j/collection1/ and leader is http://127.0.0.1:35697/wvvc/j/collection1/
   [junit4]   2> 455882 T2984 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35697/wvvc/j
   [junit4]   2> 455883 T2984 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 455883 T2984 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 455883 T2984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 455885 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455885 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455885 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455885 T2984 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 455886 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 455887 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35697/wvvc/j",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35697_wvvc%2Fj",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 455888 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 455990 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455990 T2979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 455990 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 456013 T2934 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 456014 T2934 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 456015 T2934 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59634
   [junit4]   2> 456016 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 456016 T2934 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 456016 T2934 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004
   [junit4]   2> 456017 T2934 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/'
   [junit4]   2> 456036 T2934 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/solr.xml
   [junit4]   2> 456070 T2934 oasc.CoreContainer.<init> New CoreContainer 30584360
   [junit4]   2> 456070 T2934 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/]
   [junit4]   2> 456071 T2934 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 456072 T2934 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 456072 T2934 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 456072 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 456073 T2934 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 456073 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 456073 T2934 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 456074 T2934 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 456074 T2934 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 456075 T2934 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 456075 T2934 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 456075 T2934 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 456076 T2934 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52067/solr
   [junit4]   2> 456076 T2934 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 456077 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 456078 T2996 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8018d name:ZooKeeperConnection Watcher:127.0.0.1:52067 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 456078 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 456080 T2934 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 456080 T2998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a1fb0 name:ZooKeeperConnection Watcher:127.0.0.1:52067/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 456081 T2934 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 456084 T2934 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 457087 T2934 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58454_wvvc%2Fj
   [junit4]   2> 457088 T2934 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58454_wvvc%2Fj
   [junit4]   2> 457090 T2979 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 457091 T2956 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 457090 T2998 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 457091 T2966 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 457096 T2999 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 457096 T2999 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 457097 T2999 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 457097 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 457097 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 457097 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 457098 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 457099 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58454/wvvc/j",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58454_wvvc%2Fj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 457099 T2957 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 457099 T2957 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 457101 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 457203 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 457203 T2979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 457203 T2998 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 457203 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 458097 T2999 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 458098 T2999 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 458099 T2999 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 458099 T2999 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 458100 T2999 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 458101 T2999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/collection1/'
   [junit4]   2> 458102 T2999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 458103 T2999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 458104 T2999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 458144 T2999 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 458179 T2999 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 458180 T2999 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 458189 T2999 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 458455 T2999 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 458455 T2999 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 458455 T2999 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 458461 T2999 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 458463 T2999 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 458471 T2999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 458474 T2999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 458476 T2999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 458477 T2999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 458477 T2999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 458477 T2999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 458478 T2999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 458478 T2999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 458478 T2999 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 458479 T2999 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 458479 T2999 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty2/
   [junit4]   2> 458479 T2999 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ef1d4c
   [junit4]   2> 458480 T2999 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty2
   [junit4]   2> 458480 T2999 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty2/index/
   [junit4]   2> 458480 T2999 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 458480 T2999 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-001/jetty2/index
   [junit4]   2> 458481 T2999 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=14.2666015625, floorSegmentMB=0.66015625, forceMergeDeletesPctAllowed=9.833457138872769, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 458481 T2999 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@bee3fb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@776780),segFN=segments_1,generation=1}
   [junit4]   2> 458481 T2999 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 458484 T2999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 458484 T2999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 458484 T2999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 458485 T2999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 458485 T2999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 458485 T2999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 458485 T2999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 458486 T2999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 458486 T2999 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 458486 T2999 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 458487 T2999 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 458487 T2999 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 458487 T2999 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 458488 T2999 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 458488 T2999 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 458489 T2999 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 458491 T2999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 458493 T2999 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 458493 T2999 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 458494 T2999 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 458494 T2999 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@bee3fb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@776780),segFN=segments_1,generation=1}
   [junit4]   2> 458495 T2999 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 458495 T2999 oass.SolrIndexSearcher.<init> Opening Searcher@1dfc687[collection1] main
   [junit4]   2> 458495 T2999 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 458496 T2999 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 458496 T2999 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 458497 T2999 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 458497 T2999 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 458497 T2999 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 458497 T2999 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 458498 T2999 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 458498 T2999 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 458498 T2999 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 458501 T3000 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dfc687[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 458503 T2999 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 458504 T3003 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58454/wvvc/j collection:collection1 shard:shard1
   [junit4]   2> 458505 T2934 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 458505 T2934 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 458505 T3003 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 458506 T2934 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 458507 T2934 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 458507 T2934 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 458508 T2934 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 458509 T3003 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 458510 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458510 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458510 T3003 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 458510 T3003 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 458510 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C416 name=collection1 org.apache.solr.core.SolrCore@32d8d5 url=http://127.0.0.1:58454/wvvc/j/collection1 node=127.0.0.1:58454_wvvc%2Fj C416_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:58454/wvvc/j, core=collection1, node_name=127.0.0.1:58454_wvvc%2Fj}
   [junit4]   2> 458511 T3003 C416 P58454 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58454/wvvc/j/collection1/
   [junit4]   2> 458511 T3003 C416 P58454 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 458511 T3003 C416 P58454 oasc.SyncStrategy.syncToMe http://127.0.0.1:58454/wvvc/j/collection1/ has no replicas
   [junit4]   2> 458511 T3003 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58454/wvvc/j/collection1/ shard1
   [junit4]   2> 458512 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 458512 T3003 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 458513 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458514 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458516 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458617 T2979 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 458617 T2966 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 458617 T2998 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 458617 T2956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 458665 T3003 oasc.ZkController.register We are http://127.0.0.1:58454/wvvc/j/collection1/ and leader is http://127.0.0.1:58454/wvvc/j/collection1/
   [junit4]   2> 458665 T3003 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58454/wvvc/j
   [junit4]   2> 458666 T3003 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 458666 T3003 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 458666 T3003 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 458667 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458667 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458667 T3003 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 458668 T2956 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 458669 T2957 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 458669 T2957 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"ht

[...truncated too long message...]

rStateUpdater.checkIfIamStillLeader(Overseer.java:316)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:288)
   [junit4]   2> 
   [junit4]   2> ASYNC  NEW_CORE C442 name=c8n_1x3_shard1_replica3 org.apache.solr.core.SolrCore@5758c0 url=http://127.0.0.1:58454/wvvc/j/c8n_1x3_shard1_replica3 node=127.0.0.1:58454_wvvc%2Fj C442_STATE=coll:c8n_1x3 core:c8n_1x3_shard1_replica3 props:{state=active, base_url=http://127.0.0.1:58454/wvvc/j, core=c8n_1x3_shard1_replica3, node_name=127.0.0.1:58454_wvvc%2Fj}
   [junit4]   2> 565284 T2998 C442 P58454 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 565285 T2998 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 565285 T2998 oasc.SolrCore.close [c8n_1x3_shard1_replica3]  CLOSING SolrCore org.apache.solr.core.SolrCore@5758c0
   [junit4]   2> 565307 T2998 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=302,transaction_logs_total_number=2}
   [junit4]   2> 565308 T2998 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 565308 T2998 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 565308 T2998 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 565309 T2998 C442 P58454 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
   [junit4]   2> 565313 T2998 C442 P58454 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@173eba1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12e001),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@173eba1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12e001),segFN=segments_4,generation=4}
   [junit4]   2> 565314 T2998 C442 P58454 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 565315 T2998 oasc.SolrCore.closeSearcher [c8n_1x3_shard1_replica3] Closing main searcher on request.
   [junit4]   2> 565315 T2998 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 565315 T2998 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data [CachedDir<<refCount=0;path=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data;done=false>>]
   [junit4]   2> 565315 T2998 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data
   [junit4]   2> 565315 T2998 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data/index.20140713024722199 [CachedDir<<refCount=0;path=./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data/index.20140713024722199;done=false>>]
   [junit4]   2> 565316 T2998 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001/tempDir-004/c8n_1x3_shard1_replica3/data/index.20140713024722199
   [junit4]   2> 565316 T2998 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/c8n_1x3_lf/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 565317 T2998 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 565318 T2998 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-6A6ED270C4379229-001
   [junit4]   2> NOTE: test params are: codec=Lucene49: {range_facet_l=PostingsFormat(name=FSTOrdPulsing41), multiDefault=PostingsFormat(name=FSTOrdPulsing41), intDefault=FSTOrd41, text=PostingsFormat(name=Memory doPackFST= false), range_facet_si=PostingsFormat(name=FSTOrdPulsing41), timestamp=PostingsFormat(name=FSTOrdPulsing41), range_facet_sl=Lucene41(blocksize=128), _version_=FSTOrd41, id=Lucene41(blocksize=128), a_t=PostingsFormat(name=FSTOrdPulsing41)}, docValues:{}, sim=DefaultSimilarity, locale=es_GT, timezone=America/Catamarca
   [junit4]   2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=203215784,total=282329088
   [junit4]   2> NOTE: All tests run in this JVM: [RollingRestartTest, TestNumberUtils, TestSort, PreAnalyzedUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestCollapseQParserPlugin, UpdateParamsTest, SignatureUpdateProcessorFactoryTest, PluginInfoTest, CoreAdminCreateDiscoverTest, SuggesterWFSTTest, PrimitiveFieldTypeTest, TestCollationFieldDocValues, HdfsSyncSliceTest, TestMaxScoreQueryParser, TestSchemaResource, TestRandomDVFaceting, BasicFunctionalityTest, TestCopyFieldCollectionResource, DeleteShardTest, TestStressUserVersions, CoreAdminHandlerTest, DateMathParserTest, TestReplicationHandlerBackup, CircularListTest, LeaderElectionTest, TestStressVersions, StandardRequestHandlerTest, RequiredFieldsTest, TestReload, XsltUpdateRequestHandlerTest, CoreMergeIndexesAdminHandlerTest, MoreLikeThisHandlerTest, TriLevelCompositeIdRoutingTest, TestReplicationHandler, TestAnalyzedSuggestions, TestFieldTypeResource, FastVectorHighlighterTest, HdfsUnloadDistributedZkTest, DateFieldTest, SuggesterTest, CursorMarkTest, HttpPartitionTest]
   [junit4] Completed on J0 in 114.57s, 1 test, 1 failure <<< FAILURES!

[...truncated 949 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 1 failure, 37 ignored (16 assumptions)

Total time: 73 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_20-ea-b21) - Build # 10681 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10681/
Java: 64bit/jdk1.8.0_20-ea-b21 -XX:-UseCompressedOops -XX:+UseG1GC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
ERROR: SolrZkClient opens=20 closes=19

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=20 closes=19
	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:438)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:183)
	at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    2) Thread[id=4730, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   2) Thread[id=4730, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
some core start times did not change on reload

Stack Trace:
java.lang.AssertionError: some core start times did not change on reload
	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC:E1AC349800E634F0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:838)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"state":"active","base_url":"http://127.0.0.1:42660","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:42660_","leader":"true"}]; clusterState: live nodes:[127.0.0.1:42660_, 127.0.0.1:56034_] collections:{control_collection=DocCollection(control_collection)={   "replicationFactor":"1",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:42660",           "core":"collection1",           "node_name":"127.0.0.1:42660_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={   "replicationFactor":"1",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{"core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:35580",           "core":"collection1",           "node_name":"127.0.0.1:35580_",           "leader":"true"}}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{"core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:56034",           "core":"collection1",           "node_name":"127.0.0.1:56034_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "autoCreated":"true",   "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:35580",           "core":"c8n_1x2_shard1_replica1",           "node_name":"127.0.0.1:35580_"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:56034",           "core":"c8n_1x2_shard1_replica2",           "node_name":"127.0.0.1:56034_",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"active",           "base_url":"http://127.0.0.1:56034",           "core":"c8n_1x3_shard1_replica1",           "node_name":"127.0.0.1:56034_",           "leader":"true"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:42660",           "core":"c8n_1x3_shard1_replica2",           "node_name":"127.0.0.1:42660_"},         "core_node3":{           "state":"down",           "base_url":"http://127.0.0.1:35580",           "core":"c8n_1x3_shard1_replica3",           "node_name":"127.0.0.1:35580_"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:56034",           "core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:56034_"},         "core_node2":{           "state":"active",           "base_url":"http://127.0.0.1:42660",           "core":"c8n_1x3_lf_shard1_replica3",           "node_name":"127.0.0.1:42660_",           "leader":"true"},         "core_node3":{           "state":"down",           "base_url":"http://127.0.0.1:35580",           "core":"c8n_1x3_lf_shard1_replica2",           "node_name":"127.0.0.1:35580_"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1"}}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"state":"active","base_url":"http://127.0.0.1:42660","core":"c8n_1x3_lf_shard1_replica3","node_name":"127.0.0.1:42660_","leader":"true"}]; clusterState: live nodes:[127.0.0.1:42660_, 127.0.0.1:56034_] collections:{control_collection=DocCollection(control_collection)={
  "replicationFactor":"1",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:42660",
          "core":"collection1",
          "node_name":"127.0.0.1:42660_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, collection1=DocCollection(collection1)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{"core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:35580",
          "core":"collection1",
          "node_name":"127.0.0.1:35580_",
          "leader":"true"}}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{"core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:56034",
          "core":"collection1",
          "node_name":"127.0.0.1:56034_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "autoCreated":"true",
  "maxShardsPerNode":"1"}, c8n_1x2=DocCollection(c8n_1x2)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:35580",
          "core":"c8n_1x2_shard1_replica1",
          "node_name":"127.0.0.1:35580_"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:56034",
          "core":"c8n_1x2_shard1_replica2",
          "node_name":"127.0.0.1:56034_",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3=DocCollection(c8n_1x3)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"active",
          "base_url":"http://127.0.0.1:56034",
          "core":"c8n_1x3_shard1_replica1",
          "node_name":"127.0.0.1:56034_",
          "leader":"true"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:42660",
          "core":"c8n_1x3_shard1_replica2",
          "node_name":"127.0.0.1:42660_"},
        "core_node3":{
          "state":"down",
          "base_url":"http://127.0.0.1:35580",
          "core":"c8n_1x3_shard1_replica3",
          "node_name":"127.0.0.1:35580_"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}, c8n_1x3_lf=DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:56034",
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:56034_"},
        "core_node2":{
          "state":"active",
          "base_url":"http://127.0.0.1:42660",
          "core":"c8n_1x3_lf_shard1_replica3",
          "node_name":"127.0.0.1:42660_",
          "leader":"true"},
        "core_node3":{
          "state":"down",
          "base_url":"http://127.0.0.1:35580",
          "core":"c8n_1x3_lf_shard1_replica2",
          "node_name":"127.0.0.1:35580_"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1"}}
	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC:E1AC349800E634F0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:368)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11145 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/init-core-data-001
   [junit4]   2> 305115 T779 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 305118 T779 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 305118 T779 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 305119 T780 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 305219 T779 oasc.ZkTestServer.run start zk server on port:41004
   [junit4]   2> 305220 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 305222 T786 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f67c72f name:ZooKeeperConnection Watcher:127.0.0.1:41004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 305223 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 305223 T779 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 305226 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 305227 T788 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@255809a3 name:ZooKeeperConnection Watcher:127.0.0.1:41004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 305227 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 305227 T779 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 305229 T779 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 305230 T779 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 305231 T779 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 305233 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 305233 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 305235 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 305235 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 305237 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 305237 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 305238 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 305239 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 305244 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 305244 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 305246 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 305246 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 305247 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 305248 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 305249 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 305249 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 305251 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 305251 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 305252 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 305253 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 305254 T779 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 305254 T779 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 305586 T779 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 305591 T779 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48757
   [junit4]   2> 305591 T779 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 305592 T779 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 305592 T779 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002
   [junit4]   2> 305592 T779 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/'
   [junit4]   2> 305605 T779 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/solr.xml
   [junit4]   2> 305614 T779 oasc.CoreContainer.<init> New CoreContainer 1425717133
   [junit4]   2> 305614 T779 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/]
   [junit4]   2> 305615 T779 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 305615 T779 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 305615 T779 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 305615 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 305615 T779 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 305616 T779 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 305616 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 305616 T779 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 305616 T779 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 305617 T779 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 305617 T779 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 305617 T779 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 305617 T779 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41004/solr
   [junit4]   2> 305618 T779 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 305618 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 305620 T799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28f5f2e4 name:ZooKeeperConnection Watcher:127.0.0.1:41004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 305620 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 305622 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 305628 T801 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4907dd65 name:ZooKeeperConnection Watcher:127.0.0.1:41004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 305628 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 305630 T779 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 305633 T779 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 305635 T779 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 305637 T779 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 305638 T779 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 305640 T779 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 305640 T779 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42660_
   [junit4]   2> 305641 T779 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42660_
   [junit4]   2> 305642 T779 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 305643 T779 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 305645 T779 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42660_
   [junit4]   2> 305645 T779 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 305646 T779 oasc.Overseer.start Overseer (id=92092634534445059-127.0.0.1:42660_-n_0000000000) starting
   [junit4]   2> 305647 T779 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 305651 T803 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 305651 T779 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 305653 T779 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 305654 T779 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 305655 T802 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 305656 T804 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 305656 T804 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 305657 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 305657 T804 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 305658 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 305658 T802 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42660",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42660_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 305658 T802 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 305659 T802 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 305660 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 305660 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 306657 T804 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 306658 T804 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 306658 T804 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 306659 T804 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 306659 T804 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 306660 T804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/collection1/'
   [junit4]   2> 306660 T804 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 306660 T804 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 306661 T804 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 306678 T804 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 306688 T804 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 306689 T804 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 306694 T804 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 306834 T804 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 306834 T804 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 306835 T804 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 306839 T804 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 306840 T804 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 306849 T804 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 306852 T804 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 306855 T804 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 306855 T804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 306856 T804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 306856 T804 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 306857 T804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 306857 T804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 306857 T804 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 306857 T804 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 306858 T804 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/control/data/
   [junit4]   2> 306858 T804 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@34b6994e
   [junit4]   2> 306859 T804 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/control/data
   [junit4]   2> 306859 T804 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/control/data/index/
   [junit4]   2> 306859 T804 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 306860 T804 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/control/data/index
   [junit4]   2> 306860 T804 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1142995002, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 306861 T804 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75492a86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a75e2e6),segFN=segments_1,generation=1}
   [junit4]   2> 306861 T804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 306864 T804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 306865 T804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 306865 T804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 306865 T804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 306865 T804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 306866 T804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 306866 T804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 306866 T804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 306866 T804 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 306867 T804 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 306867 T804 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 306867 T804 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 306867 T804 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 306868 T804 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 306868 T804 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 306868 T804 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 306870 T804 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 306871 T804 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 306871 T804 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 306872 T804 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=41, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4067052581983801]
   [junit4]   2> 306873 T804 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75492a86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a75e2e6),segFN=segments_1,generation=1}
   [junit4]   2> 306873 T804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 306873 T804 oass.SolrIndexSearcher.<init> Opening Searcher@11678c82[collection1] main
   [junit4]   2> 306873 T804 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 306874 T804 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 306874 T804 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 306875 T804 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 306875 T804 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 306875 T804 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 306875 T804 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 306876 T804 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 306876 T804 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 306876 T804 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 306877 T805 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11678c82[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 306878 T804 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 306879 T808 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42660 collection:control_collection shard:shard1
   [junit4]   2> 306879 T779 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 306879 T808 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 306879 T779 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 306883 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 306883 T808 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 306884 T811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f590f66 name:ZooKeeperConnection Watcher:127.0.0.1:41004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 306884 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 306885 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 306885 T779 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 306885 T808 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 306885 T808 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C204 name=collection1 org.apache.solr.core.SolrCore@6fb3264d url=http://127.0.0.1:42660/collection1 node=127.0.0.1:42660_ C204_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:42660, core=collection1, node_name=127.0.0.1:42660_}
   [junit4]   2> 306885 T808 C204 P42660 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42660/collection1/
   [junit4]   2> 306886 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 306886 T808 C204 P42660 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 306886 T808 C204 P42660 oasc.SyncStrategy.syncToMe http://127.0.0.1:42660/collection1/ has no replicas
   [junit4]   2> 306886 T808 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42660/collection1/ shard1
   [junit4]   2> 306886 T779 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 306886 T808 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 306887 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 306889 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 306891 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 306992 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 306992 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 307040 T808 oasc.ZkController.register We are http://127.0.0.1:42660/collection1/ and leader is http://127.0.0.1:42660/collection1/
   [junit4]   2> 307040 T808 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42660
   [junit4]   2> 307040 T808 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 307041 T808 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 307041 T808 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 307042 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307042 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307042 T808 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 307042 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307044 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 307044 T802 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42660",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42660_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 307045 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307116 T779 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 307117 T779 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 307119 T779 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37277
   [junit4]   2> 307119 T779 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 307120 T779 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 307120 T779 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003
   [junit4]   2> 307120 T779 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/'
   [junit4]   2> 307131 T779 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/solr.xml
   [junit4]   2> 307139 T779 oasc.CoreContainer.<init> New CoreContainer 323516013
   [junit4]   2> 307140 T779 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/]
   [junit4]   2> 307140 T779 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 307140 T779 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 307141 T779 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 307141 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 307141 T779 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 307141 T779 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 307141 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 307142 T779 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 307142 T779 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 307142 T779 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 307143 T779 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 307143 T779 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 307143 T779 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41004/solr
   [junit4]   2> 307143 T779 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 307144 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 307146 T822 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cc89338 name:ZooKeeperConnection Watcher:127.0.0.1:41004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 307146 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 307146 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 307146 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 307148 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 307150 T824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f28a5a3 name:ZooKeeperConnection Watcher:127.0.0.1:41004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 307150 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 307153 T779 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 308155 T779 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56034_
   [junit4]   2> 308156 T779 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56034_
   [junit4]   2> 308158 T801 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 308158 T811 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 308158 T824 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 308165 T825 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 308165 T825 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 308165 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 308166 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 308166 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 308165 T825 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 308167 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 308167 T802 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56034",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56034_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 308167 T802 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 308168 T802 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 308169 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 308271 T824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 308271 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 308271 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309166 T825 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 309166 T825 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 309167 T825 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 309167 T825 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 309168 T825 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 309168 T825 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/collection1/'
   [junit4]   2> 309169 T825 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 309169 T825 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 309169 T825 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 309188 T825 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 309197 T825 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 309198 T825 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 309203 T825 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 309310 T825 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 309310 T825 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 309311 T825 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 309315 T825 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 309316 T825 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 309327 T825 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309329 T825 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309331 T825 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309332 T825 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309332 T825 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309332 T825 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309333 T825 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309333 T825 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309333 T825 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 309334 T825 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 309334 T825 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty1/
   [junit4]   2> 309334 T825 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@34b6994e
   [junit4]   2> 309335 T825 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty1
   [junit4]   2> 309335 T825 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty1/index/
   [junit4]   2> 309335 T825 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 309335 T825 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty1/index
   [junit4]   2> 309336 T825 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1142995002, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 309337 T825 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@502024c5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2096a3eb),segFN=segments_1,generation=1}
   [junit4]   2> 309337 T825 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309340 T825 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 309340 T825 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 309341 T825 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 309341 T825 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309341 T825 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309341 T825 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 309341 T825 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309342 T825 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309342 T825 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 309342 T825 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 309343 T825 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 309343 T825 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 309343 T825 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 309343 T825 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 309344 T825 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 309344 T825 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 309346 T825 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 309347 T825 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 309347 T825 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 309348 T825 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=41, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4067052581983801]
   [junit4]   2> 309349 T825 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@502024c5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2096a3eb),segFN=segments_1,generation=1}
   [junit4]   2> 309349 T825 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309349 T825 oass.SolrIndexSearcher.<init> Opening Searcher@597910c5[collection1] main
   [junit4]   2> 309349 T825 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 309350 T825 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 309350 T825 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 309350 T825 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 309350 T825 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 309351 T825 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 309351 T825 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 309351 T825 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 309351 T825 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 309351 T825 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 309354 T826 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@597910c5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 309355 T825 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 309356 T829 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56034 collection:collection1 shard:shard2
   [junit4]   2> 309356 T779 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 309357 T829 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 309357 T779 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 309360 T829 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 309361 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309361 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309361 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309362 T829 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 309362 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 309362 T829 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C205 name=collection1 org.apache.solr.core.SolrCore@16eeef81 url=http://127.0.0.1:56034/collection1 node=127.0.0.1:56034_ C205_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:56034, core=collection1, node_name=127.0.0.1:56034_}
   [junit4]   2> 309363 T829 C205 P56034 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56034/collection1/
   [junit4]   2> 309363 T829 C205 P56034 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 309363 T829 C205 P56034 oasc.SyncStrategy.syncToMe http://127.0.0.1:56034/collection1/ has no replicas
   [junit4]   2> 309363 T829 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56034/collection1/ shard2
   [junit4]   2> 309363 T829 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 309364 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309366 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309368 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309469 T824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309469 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309469 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309517 T829 oasc.ZkController.register We are http://127.0.0.1:56034/collection1/ and leader is http://127.0.0.1:56034/collection1/
   [junit4]   2> 309517 T829 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56034
   [junit4]   2> 309518 T829 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 309518 T829 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 309518 T829 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 309519 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309519 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309519 T829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 309519 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309520 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 309521 T802 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56034",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56034_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 309522 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 309599 T779 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 309600 T779 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 309602 T779 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38974
   [junit4]   2> 309603 T779 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 309603 T779 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 309603 T779 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004
   [junit4]   2> 309604 T779 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/'
   [junit4]   2> 309624 T824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309624 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309624 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 309625 T779 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/solr.xml
   [junit4]   2> 309634 T779 oasc.CoreContainer.<init> New CoreContainer 783017830
   [junit4]   2> 309634 T779 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/]
   [junit4]   2> 309635 T779 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 309635 T779 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 309635 T779 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 309635 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 309636 T779 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 309636 T779 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 309636 T779 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 309636 T779 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 309637 T779 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 309637 T779 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 309637 T779 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 309638 T779 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 309638 T779 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41004/solr
   [junit4]   2> 309638 T779 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 309639 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 309640 T841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@362ecd3a name:ZooKeeperConnection Watcher:127.0.0.1:41004 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309641 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 309644 T779 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 309645 T843 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5445c1f9 name:ZooKeeperConnection Watcher:127.0.0.1:41004/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 309646 T779 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 309650 T779 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 310651 T779 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35580_
   [junit4]   2> 310652 T779 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35580_
   [junit4]   2> 310654 T811 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 310654 T843 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 310654 T824 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 310654 T801 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 310658 T844 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 310658 T844 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 310658 T844 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 310658 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 310659 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 310659 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 310660 T802 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 310660 T802 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35580",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35580_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 310660 T802 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 310660 T802 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 310661 T801 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 310763 T811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 310763 T843 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 310763 T801 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 310763 T824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 311659 T844 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 311659 T844 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 311659 T844 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 311659 T844 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 311660 T844 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 311660 T844 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/collection1/'
   [junit4]   2> 311661 T844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 311661 T844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 311661 T844 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 311679 T844 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 311689 T844 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 311690 T844 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 311694 T844 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 311802 T844 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 311802 T844 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 311803 T844 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 311807 T844 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 311808 T844 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 311815 T844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 311818 T844 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 311820 T844 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 311820 T844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 311820 T844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 311820 T844 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 311821 T844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 311821 T844 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 311821 T844 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 311821 T844 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 311822 T844 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty2/
   [junit4]   2> 311822 T844 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@34b6994e
   [junit4]   2> 311823 T844 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty2
   [junit4]   2> 311823 T844 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty2/index/
   [junit4]   2> 311823 T844 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 311823 T844 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-604ABA8077B954CC-001/tempDir-001/jetty2/index
   [junit4]   2> 311824 T844 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1142995002, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 311824 T844 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4274e96f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2302f98b),segFN=segments_1,generation=1}
   [junit4]   2> 311824 T844 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 311827 T844 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 311827 T844 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 311827 T844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 311827 T844 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 311828 T844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 311828 T844 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 311828 T844 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 311828 T844 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 311828 T844 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 311829 T844 oasc.RequestHandlers.initHandlersFromConfig creat

[...truncated too long message...]

> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 994314 T4729 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]   2> 
   [junit4]   2> 994814 T4729 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 994815 T4729 oaz.ClientCnxn$SendThread.run WARN Session 0x1472dcbe7f50010 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 995314 T4729 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 995316 T4729 oaz.ClientCnxn$SendThread.run WARN Session 0x1472dcbe7f50010 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 131503 T4569 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=lv_LV, timezone=Asia/Aqtau
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_20-ea (64-bit)/cpus=8,threads=2,free=97943136,total=475004928
   [junit4]   2> NOTE: All tests run in this JVM: [DirectSolrConnectionTest, TestBadConfig, InfoHandlerTest, DistributedQueryComponentCustomSortTest, TestXIncludeConfig, TestCSVLoader, SuggesterFSTTest, TestPerFieldSimilarity, TestPhraseSuggestions, ConnectionManagerTest, TestInfoStreamLogging, CopyFieldTest, LeaderElectionIntegrationTest, EnumFieldTest, LegacyHTMLStripCharFilterTest, TestSchemaVersionResource, ChaosMonkeyNothingIsSafeTest, TestJoin, TestNonDefinedSimilarityFactory, TestLRUCache, QueryParsingTest, AnalyticsQueryTest, SimplePostToolTest, TestQueryUtils, TestSystemIdResolver, AnalyticsMergeStrategyTest, TestFieldResource, TestFastLRUCache, SchemaVersionSpecificBehaviorTest, TestCollationField, TestIBSimilarityFactory, TestSolrQueryParserResource, TestSerializedLuceneMatchVersion, AnalysisAfterCoreReloadTest, TestSolrXmlPersistence, ZkSolrClientTest, TestPostingsSolrHighlighter, TestExceedMaxTermLength, SoftAutoCommitTest, FullSolrCloudDistribCmdsTest, TestSolrJ, TestSuggestSpellingConverter, TestDistributedGrouping, OpenCloseCoreStressTest, PreAnalyzedFieldTest, CacheHeaderTest, TimeZoneUtilsTest, AtomicUpdatesTest, AliasIntegrationTest, SearchHandlerTest, DistributedQueryElevationComponentTest, AlternateDirectoryTest, TestZkChroot, TestCollapseQParserPlugin, TestRandomMergePolicy, TriLevelCompositeIdRoutingTest, BlockCacheTest, TestSolrXmlPersistor, TestNonNRTOpen, DocExpirationUpdateProcessorFactoryTest, DebugComponentTest, TestSolrQueryParser, TestUniqueKeyFieldResource, TestStandardQParsers, TestUpdate, TestCSVResponseWriter, RankQueryTest, TestFunctionQuery, DistanceFunctionTest, TestCollectionAPI, TestQuerySenderNoQuery, CSVRequestHandlerTest, ChangedSchemaMergeTest, DistributedTermsComponentTest, MultiThreadedOCPTest, QueryElevationComponentTest, BinaryUpdateRequestHandlerTest, TestBinaryField, TestLazyCores, DistributedQueryComponentOptimizationTest, TestSweetSpotSimilarityFactory, MigrateRouteKeyTest, TestReloadAndDeleteDocs, DistributedDebugComponentTest, TestMergePolicyConfig, RequestHandlersTest, ScriptEngineTest, TestSchemaSimilarityResource, TestValueSourceCache, TestNoOpRegenerator, CoreContainerCoreInitFailuresTest, TestElisionMultitermQuery, TestLeaderElectionZkExpiry, SolrRequestParserTest, IndexSchemaRuntimeFieldTest, DistribDocExpirationUpdateProcessorTest, TestWriterPerf, SolrInfoMBeanTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.seed=604ABA8077B954CC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lv_LV -Dtests.timezone=Asia/Aqtau -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | CollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=20 closes=19
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:438)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:183)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   [junit4]    >    1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    2) Thread[id=4730, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=4729, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[604ABA8077B954CC]-SendThread(127.0.0.1:51049), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([604ABA8077B954CC]:0)
   [junit4] Completed on J1 in 131.56s, 1 test, 2 failures, 2 errors <<< FAILURES!

[...truncated 605 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 3 suite-level errors, 2 failures, 698 ignored (16 assumptions)

Total time: 56 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20-ea-b21 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any