You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/29 11:34:32 UTC

[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b04) - Build # 4343 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4343/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"c8n_1x2_leader_session_loss_shard1_replica1",           "base_url":"http://127.0.0.1:57338/iw_hj",           "node_name":"127.0.0.1:57338_iw_hj",           "state":"active",           "leader":"true"},         "core_node2":{           "core":"c8n_1x2_leader_session_loss_shard1_replica2",           "base_url":"http://127.0.0.1:57322/iw_hj",           "node_name":"127.0.0.1:57322_iw_hj",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x2_leader_session_loss_shard1_replica1",
          "base_url":"http://127.0.0.1:57338/iw_hj",
          "node_name":"127.0.0.1:57338_iw_hj",
          "state":"active",
          "leader":"true"},
        "core_node2":{
          "core":"c8n_1x2_leader_session_loss_shard1_replica2",
          "base_url":"http://127.0.0.1:57322/iw_hj",
          "node_name":"127.0.0.1:57322_iw_hj",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([299406B9921B386F:A87288A1E5445853]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:56139, https://127.0.0.1:56148, https://127.0.0.1:56124, https://127.0.0.1:56157, https://127.0.0.1:56166]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:56139, https://127.0.0.1:56148, https://127.0.0.1:56124, https://127.0.0.1:56157, https://127.0.0.1:56166]
	at __randomizedtesting.SeedInfo.seed([299406B9921B386F:A87288A1E5445853]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:171)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:144)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard1/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 47 more




Build Log:
[...truncated 10950 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\init-core-data-001
   [junit4]   2> 1137521 T3132 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1137522 T3132 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1137526 T3132 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1137528 T3132 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1137529 T3133 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1137640 T3132 oasc.ZkTestServer.run start zk server on port:56114
   [junit4]   2> 1137640 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1137646 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1137652 T3139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a8820c6 name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1137652 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1137652 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1137652 T3132 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1137661 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1137664 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1137667 T3141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43a4dd62 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1137667 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1137667 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1137667 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1137673 T3132 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1137679 T3132 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1137682 T3132 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1137686 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1137686 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1137696 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1137696 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1137703 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1137703 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1137709 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1137709 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1137715 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1137715 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1137721 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1137721 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1137727 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1137727 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1137732 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1137732 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1137738 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1137738 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1137742 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1137742 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1137748 T3132 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1137748 T3132 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1137753 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1137753 T3134 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148c0738eef0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1137755 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1137756 T3143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bc5b881 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1137758 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1137759 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1137761 T3134 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148c0738eef0002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1139641 T3132 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1139652 T3132 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1139655 T3132 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:56124
   [junit4]   2> 1139659 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1139659 T3132 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1139659 T3132 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002
   [junit4]   2> 1139660 T3132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002\'
   [junit4]   2> 1139697 T3132 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002\solr.xml
   [junit4]   2> 1139707 T3132 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002
   [junit4]   2> 1139707 T3132 oasc.CoreContainer.<init> New CoreContainer 434546511
   [junit4]   2> 1139709 T3132 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002\]
   [junit4]   2> 1139710 T3132 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1139710 T3132 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1139710 T3132 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1139710 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1139712 T3132 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1139712 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1139712 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1139712 T3132 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1139712 T3132 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1139715 T3132 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1139718 T3132 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1139718 T3132 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1139718 T3132 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1139718 T3132 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56114/solr
   [junit4]   2> 1139718 T3132 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1139718 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1139723 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1139725 T3154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c35d6e0 name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1139726 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1139726 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1139729 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1139730 T3156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f81053 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1139731 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1139732 T3132 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1139738 T3132 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1139742 T3132 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1139747 T3132 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1139751 T3132 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1139756 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1139760 T3132 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56124_
   [junit4]   2> 1139761 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56124_
   [junit4]   2> 1139765 T3132 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1139770 T3132 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1139773 T3132 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1139776 T3132 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:56124_
   [junit4]   2> 1139776 T3132 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1139779 T3132 oasc.Overseer.start Overseer (id=92535394912894980-127.0.0.1:56124_-n_0000000000) starting
   [junit4]   2> 1139784 T3132 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1139799 T3132 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1139800 T3158 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1139802 T3132 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1139806 T3132 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1139808 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1139816 T3157 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1139818 T3132 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002
   [junit4]   2> 1141025 T3132 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002\conf\
   [junit4]   2> 1141241 T3132 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1141243 T3132 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1141243 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1141256 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1141261 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1141264 T3162 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6def4a16 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141264 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141264 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1141266 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1141272 T3132 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1141272 T3132 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1141272 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1141275 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1141277 T3164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@105e0133 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1141277 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1141277 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1141289 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1141293 T3157 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1141293 T3157 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1141293 T3157 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1141299 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1141301 T3165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1141306 T3166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1143225 T3132 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1143225 T3132 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1143236 T3132 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1143238 T3132 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:56139
   [junit4]   2> 1143243 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1143243 T3132 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1143243 T3132 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003
   [junit4]   2> 1143243 T3132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003\'
   [junit4]   2> 1143280 T3132 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003\solr.xml
   [junit4]   2> 1143291 T3132 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003
   [junit4]   2> 1143291 T3132 oasc.CoreContainer.<init> New CoreContainer 1770044164
   [junit4]   2> 1143291 T3132 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003\]
   [junit4]   2> 1143293 T3132 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1143293 T3132 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1143293 T3132 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1143294 T3132 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1143299 T3132 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1143302 T3132 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1143302 T3132 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1143302 T3132 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1143302 T3132 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56114/solr
   [junit4]   2> 1143302 T3132 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1143302 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1143305 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1143307 T3177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13d7267c name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1143309 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1143309 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1143311 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1143315 T3179 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2401b3f0 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1143315 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1143327 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1144485 T3132 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56139_
   [junit4]   2> 1144486 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56139_
   [junit4]   2> 1144497 T3132 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1144501 T3132 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003
   [junit4]   2> 1145232 T3132 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003\conf\
   [junit4]   2> 1145362 T3132 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1145365 T3132 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1145365 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1147420 T3132 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1147422 T3132 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1147436 T3132 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1147439 T3132 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:56148
   [junit4]   2> 1147449 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1147449 T3132 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1147450 T3132 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004
   [junit4]   2> 1147450 T3132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\'
   [junit4]   2> 1147504 T3132 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\solr.xml
   [junit4]   2> 1147521 T3132 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004
   [junit4]   2> 1147522 T3132 oasc.CoreContainer.<init> New CoreContainer 1087960447
   [junit4]   2> 1147522 T3132 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\]
   [junit4]   2> 1147524 T3132 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1147524 T3132 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1147524 T3132 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1147525 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1147525 T3132 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1147525 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1147525 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1147526 T3132 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1147526 T3132 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1147532 T3132 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1147538 T3132 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1147539 T3132 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1147539 T3132 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1147539 T3132 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56114/solr
   [junit4]   2> 1147539 T3132 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1147540 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1147544 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1147548 T3192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fd57f0b name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1147549 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1147549 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1147555 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1147560 T3194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ad57e8a name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1147560 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1147579 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1148826 T3132 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56148_
   [junit4]   2> 1148827 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56148_
   [junit4]   2> 1148838 T3132 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1148848 T3132 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004
   [junit4]   2> 1150158 T3132 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\conf\
   [junit4]   2> 1150379 T3132 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1150381 T3132 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1150381 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1152648 T3132 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1152648 T3132 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1152658 T3132 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1152660 T3132 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:56157
   [junit4]   2> 1152664 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1152664 T3132 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1152664 T3132 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005
   [junit4]   2> 1152666 T3132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005\'
   [junit4]   2> 1152701 T3132 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005\solr.xml
   [junit4]   2> 1152710 T3132 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005
   [junit4]   2> 1152710 T3132 oasc.CoreContainer.<init> New CoreContainer 1646978812
   [junit4]   2> 1152713 T3132 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005\]
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1152714 T3132 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1152720 T3132 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1152722 T3132 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1152722 T3132 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1152722 T3132 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1152722 T3132 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56114/solr
   [junit4]   2> 1152722 T3132 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1152722 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1152727 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1152730 T3207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3738533c name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1152731 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1152731 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1152736 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1152738 T3209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@568fca6 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1152738 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1152750 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1153997 T3132 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56157_
   [junit4]   2> 1154001 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56157_
   [junit4]   2> 1154011 T3132 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1154020 T3132 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005
   [junit4]   2> 1154801 T3132 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005\conf\
   [junit4]   2> 1154931 T3132 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1154934 T3132 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1154934 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1157070 T3132 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1157071 T3132 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1157082 T3132 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1157084 T3132 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:56166
   [junit4]   2> 1157090 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1157091 T3132 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1157091 T3132 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006
   [junit4]   2> 1157091 T3132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006\'
   [junit4]   2> 1157133 T3132 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006\solr.xml
   [junit4]   2> 1157149 T3132 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006
   [junit4]   2> 1157151 T3132 oasc.CoreContainer.<init> New CoreContainer 1727187013
   [junit4]   2> 1157151 T3132 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006\]
   [junit4]   2> 1157152 T3132 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1157152 T3132 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1157152 T3132 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1157152 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1157153 T3132 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1157153 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1157153 T3132 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1157153 T3132 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1157155 T3132 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1157160 T3132 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1157164 T3132 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1157164 T3132 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1157165 T3132 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1157165 T3132 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56114/solr
   [junit4]   2> 1157165 T3132 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1157167 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1157169 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1157172 T3222 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f111430 name:ZooKeeperConnection Watcher:127.0.0.1:56114 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1157173 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1157173 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1157176 T3134 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148c0738eef000d, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1157177 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1157180 T3224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5729306c name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1157180 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1157196 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1158569 T3132 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56166_
   [junit4]   2> 1158571 T3132 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56166_
   [junit4]   2> 1158593 T3132 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1158599 T3132 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006
   [junit4]   2> 1160154 T3132 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-006\conf\
   [junit4]   2> 1160407 T3132 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1160410 T3132 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1160410 T3132 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1160428 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1160431 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1160433 T3228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@456159f5 name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1160433 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1160433 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1160436 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1160443 T3132 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
   [junit4]   2> 1160444 T3132 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1160445 T3132 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1160447 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3381a1db name:ZooKeeperConnection Watcher:127.0.0.1:56114/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1160448 T3132 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1160448 T3132 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1160449 T3132 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1160538 T3149 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=delLiveColl&action=CREATE&numShards=2&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 1160539 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160542 T3158 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1160543 T3231 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1160551 T3231 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1160551 T3231 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl 
   [junit4]   2> 1160553 T3231 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1160562 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160570 T3157 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1160571 T3157 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1160571 T3157 oasc.Overseer$ClusterStateUpdater.createCollection state version delLiveColl 2
   [junit4]   2> 1160577 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/delLiveColl/state.json
   [junit4]   2> 1160581 T3165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160581 T3180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160581 T3195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160582 T3210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160581 T3225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160583 T3166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160587 T3233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160589 T3232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1160699 T3231 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1160699 T3231 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:56148_
   [junit4]   2> 1160702 T3231 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:56124_
   [junit4]   2> 1160708 T3231 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:56139_
   [junit4]   2> 1160750 T3231 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:56157_
   [junit4]   2> 1160782 T3150 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1160790 T3189 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1160840 T3171 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1160840 T3200 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1160844 T3150 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
   [junit4]   2> 1160844 T3189 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
   [junit4]   2> 1160852 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160855 T3171 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
   [junit4]   2> 1160855 T3200 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
   [junit4]   2> 1160856 T3150 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1160856 T3150 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1160858 T3189 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1160858 T3189 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1160859 T3150 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1160860 T3189 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1160864 T3171 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1160864 T3171 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1160865 T3189 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1160865 T3189 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1160865 T3171 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1160867 T3200 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1160867 T3200 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1160867 T3150 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1160868 T3150 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1160868 T3200 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1160871 T3171 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1160871 T3171 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1160873 T3200 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1160873 T3200 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1160879 T3157 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1160879 T3157 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:56124",
   [junit4]   2> 	  "node_name":"127.0.0.1:56124_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1160884 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1160887 T3165 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160887 T3180 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160887 T3195 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160888 T3210 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160889 T3180 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1160889 T3195 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1160889 T3210 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1160893 T3165 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1160903 T3157 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1160904 T3157 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:56148",
   [junit4]   2> 	  "node_name":"127.0.0.1:56148_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1160908 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160909 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1160911 T3180 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160912 T3210 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160912 T3195 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160915 T3180 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1160915 T3210 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1160916 T3165 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160916 T3195 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1160924 T3165 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1160933 T3157 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1160933 T3157 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:56139",
   [junit4]   2> 	  "node_name":"127.0.0.1:56139_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1160938 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160938 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1160940 T3180 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160940 T3165 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160941 T3195 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160941 T3210 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160942 T3180 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1160942 T3195 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1160944 T3210 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1160948 T3165 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1160957 T3157 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1160957 T3157 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:56157",
   [junit4]   2> 	  "node_name":"127.0.0.1:56157_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1160962 T3165 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1160964 T3157 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1160965 T3180 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160965 T3165 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160965 T3195 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160966 T3210 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1160968 T3180 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1160969 T3210 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1160969 T3195 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1160973 T3165 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1162224 T3189 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1162224 T3189 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1162227 T3150 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1162227 T3150 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1162229 T3189 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1162232 T3189 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1162228 T3171 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1162235 T3200 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1162236 T3200 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1162235 T3171 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1162235 T3150 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1162238 T3150 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1162244 T3200 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1162245 T3200 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1162238 T3189 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1162245 T3189 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\'
   [junit4]   2> 1162244 T3171 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1162248 T3171 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1162250 T3150 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1162250 T3150 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-002\delLiveColl_shard1_replica2\'
   [junit4]   2> 1162254 T3200 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1162254 T3200 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-005\delLiveColl_shard2_replica2\'
   [junit4]   2> 1162262 T3171 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1162262 T3171 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-003\delLiveColl_shard2_replica1\'
   [junit4]   2> 1162399 T3189 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1162403 T3200 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1162419 T3189 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1162420 T3200 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1162428 T3189 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1162435 T3171 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1162439 T3189 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1162451 T3171 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1162504 T3150 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1162521 T3150 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1162522 T3200 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1162534 T3200 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1162639 T3189 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1162652 T3189 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1162653 T3189 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1162807 T3171 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1162849 T3171 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1162877 T3200 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1162881 T3200 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1162884 T3200 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1162893 T3150 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1162901 T3150 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1162986 T3171 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1162989 T3171 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1162991 T3171 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1163028 T3189 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163032 T3150 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1163035 T3189 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163038 T3150 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1163039 T3150 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1163046 T3189 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1163047 T3171 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163051 T3189 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1163051 T3189 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1163051 T3189 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1163055 T3171 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1163060 T3189 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1163060 T3189 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1163060 T3189 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using configuration from collection delLiveColl
   [junit4]   2> 1163060 T3189 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1163062 T3189 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\data\
   [junit4]   2> 1163062 T3189 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d617332
   [junit4]   2> 1163063 T3189 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\data\
   [junit4]   2> 1163063 T3189 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\data\index/
   [junit4]   2> 1163063 T3189 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 1163064 T3189 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-299406B9921B386F-001\tempDir-004\delLiveColl_shard1_replica1\data\index
   [junit4]   2> 1163065 T3189 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=38.4814453125, floorSegmentMB=1.9443359375, forceMergeDeletesPctAllowed=16.95878649422906, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1163065 T3189 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-49c24f83)),segFN=segments_1,generation=1}
   [junit4]   2> 1163065 T3189 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1163076 T3189 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1163077 T3189 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1163077 T3189 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1163077 T3189 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1163079 T3189 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1163079 T3189 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1163079 T3189 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1163080 T3189 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1163080 T3189 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1163080 T3189 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache

[...truncated too long message...]

common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:761)
   [junit4]   2> 	... 4 more
   [junit4]   2> 
   [junit4]   2> 2107339 T5044 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148c0807000000b, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2107339 T5384 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2107339 T5390 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:57358_iw_hj
   [junit4]   2> 2107357 T5042 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/iw_hj,null}
   [junit4]   2> 2112662 T5042 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2112664 T5042 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57315 57315
   [junit4]   2> 2114828 T5043 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57315 57315
   [junit4]   2> 2114831 T5043 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		3	/solr/collections/c8n_1x3/state.json
   [junit4]   2> 		3	/solr/collections/collection1/state.json
   [junit4]   2> 		3	/solr/collections/c8n_1x2_leader_session_loss/state.json
   [junit4]   2> 		2	/solr/collections/c8n_1x2/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/live_nodes
   [junit4]   2> 		4	/solr/overseer/collection-queue-work
   [junit4]   2> 		2	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> 2114831 T5042 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:57358/iw_hj
   [junit4]   2> 2114832 T5042 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:57348/iw_hj
   [junit4]   2> 2114833 T5042 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:57338/iw_hj
   [junit4]   2> 2114833 T5042 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:57322/iw_hj
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.method=testDistribSearch -Dtests.seed=299406B9921B386F -Dtests.slow=true -Dtests.locale=pl_PL -Dtests.timezone=America/Denver -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE  133s | HttpPartitionTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
   [junit4]    >   "replicationFactor":"2",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "core":"c8n_1x2_leader_session_loss_shard1_replica1",
   [junit4]    >           "base_url":"http://127.0.0.1:57338/iw_hj",
   [junit4]    >           "node_name":"127.0.0.1:57338_iw_hj",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "core":"c8n_1x2_leader_session_loss_shard1_replica2",
   [junit4]    >           "base_url":"http://127.0.0.1:57322/iw_hj",
   [junit4]    >           "node_name":"127.0.0.1:57322_iw_hj",
   [junit4]    >           "state":"down"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false"}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([299406B9921B386F:A87288A1E5445853]:0)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2114858 T5042 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.HttpPartitionTest-299406B9921B386F-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=PostingsFormat(name=SimpleText), multiDefault=FSTOrd41, a_t=PostingsFormat(name=SimpleText), intDefault=PostingsFormat(name=SimpleText), id=PostingsFormat(name=Asserting), text=FST41, range_facet_l=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=DefaultSimilarity, locale=pl_PL, timezone=America/Denver
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=2,threads=1,free=197619384,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TimeZoneUtilsTest, TestDynamicFieldCollectionResource, MoreLikeThisHandlerTest, NoCacheHeaderTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestCollapseQParserPlugin, CoreMergeIndexesAdminHandlerTest, LeaderFailoverAfterPartitionTest, UUIDFieldTest, SortByFunctionTest, HdfsSyncSliceTest, SuggesterTSTTest, TestJoin, TestRecoveryHdfs, DisMaxRequestHandlerTest, TestSchemaVersionResource, VMParamsZkACLAndCredentialsProvidersTest, TestValueSourceCache, TestMissingGroups, PathHierarchyTokenizerFactoryTest, SuggestComponentTest, MigrateRouteKeyTest, IndexBasedSpellCheckerTest, TestClassNameShortening, TestNonNRTOpen, TestMergePolicyConfig, DirectSolrSpellCheckerTest, TestCollationFieldDocValues, TestSerializedLuceneMatchVersion, HdfsChaosMonkeySafeLeaderTest, TestSort, ClusterStateTest, TestCopyFieldCollectionResource, SolrInfoMBeanTest, SliceStateTest, PreAnalyzedUpdateProcessorTest, TestRealTimeGet, TermVectorComponentDistributedTest, OverseerRolesTest, BadCopyFieldTest, TestNoOpRegenerator, DistribCursorPagingTest, CircularListTest, TestChildDocTransformer, TestDefaultSearchFieldResource, HighlighterTest, OverseerTest, LeaderElectionIntegrationTest, StressHdfsTest, AsyncMigrateRouteKeyTest, DeleteInactiveReplicaTest, BadComponentTest, TestManagedResource, TestCodecSupport, SuggesterTest, HdfsLockFactoryTest, OverseerStatusTest, TestSolrDeletionPolicy1, AtomicUpdatesTest, ShowFileRequestHandlerTest, TestSolrCoreProperties, TestAddFieldRealTimeGet, CursorPagingTest, TestWordDelimiterFilterFactory, TestSortingResponseWriter, SpellCheckComponentTest, BadIndexSchemaTest, DistributedFacetPivotSmallTest, TestDocSet, SystemInfoHandlerTest, TestSolrJ, FileBasedSpellCheckerTest, BlockDirectoryTest, TestShardHandlerFactory, HdfsCollectionsAPIDistributedZkTest, TestBinaryField, AddBlockUpdateTest, TestExceedMaxTermLength, TestCollectionAPI, TestFieldTypeCollectionResource, TestStressVersions, TestRestManager, SpellingQueryConverterTest, OverriddenZkACLAndCredentialsProvidersTest, TestDistributedMissingSort, DeleteReplicaTest, ConnectionManagerTest, StandardRequestHandlerTest, TestSweetSpotSimilarityFactory, DeleteShardTest, LeaderElectionTest, AnalysisErrorHandlingTest, ZkControllerTest, ChaosMonkeyNothingIsSafeTest, TestDistributedGrouping, DebugComponentTest, TestMaxScoreQueryParser, TestQuerySenderNoQuery, TestConfig, TestDocumentBuilder, MergeStrategyTest, TestSolr4Spatial, TestBadConfig, SSLMigrationTest, TestIndexingPerformance, PluginInfoTest, TestMiniSolrCloudCluster, TestAnalyzeInfixSuggestions, DistribDocExpirationUpdateProcessorTest, CollectionsAPIDistributedZkTest, TestQueryTypes, TestFastLRUCache, NumericFieldsTest, TestShortCircuitedRequests, CoreAdminHandlerTest, ExternalFileFieldSortTest, BasicFunctionalityTest, HttpPartitionTest]
   [junit4] Completed in 133.32s, 1 test, 1 failure <<< FAILURES!

[...truncated 985 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:524: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:472: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:61: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1359: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 1 failure, 53 ignored (31 assumptions)

Total time: 147 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any