You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/10/29 15:58:12 UTC

[JENKINS] Lucene-Solr-4.x-Windows (32bit/jdk1.6.0_45) - Build # 3328 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3328/
Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:51128, http://127.0.0.1:51110, http://127.0.0.1:51119, http://127.0.0.1:51098, http://127.0.0.1:51137]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:51128, http://127.0.0.1:51110, http://127.0.0.1:51119, http://127.0.0.1:51098, http://127.0.0.1:51137]
	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4:C5584E9D2E5378C8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:337)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:148)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:134)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:51128
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:472)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	... 45 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=107, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=RUNNABLE, group=TGRP-DeleteReplicaTest]         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273)         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255)         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=107, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=RUNNABLE, group=TGRP-DeleteReplicaTest]
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)




Build Log:
[...truncated 9224 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 15262 T17 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 15274 T17 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\solrtest-DeleteReplicaTest-1383058354533
   [junit4]   2> 15284 T17 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 15290 T18 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 15520 T17 oasc.ZkTestServer.run start zk server on port:51091
   [junit4]   2> 15803 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 15920 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1031310 name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 15921 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 15926 T17 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 16030 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 16037 T26 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fe94a name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 16038 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 16060 T17 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 16074 T17 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 16090 T17 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 16101 T17 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 16114 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 16133 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 16157 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 16158 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 16172 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16172 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 16187 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 16187 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 16199 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 16199 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 16212 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 16212 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 16225 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16225 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 16238 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16238 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 16253 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 16253 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 16267 T17 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 16267 T17 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 17254 T17 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 17492 T17 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51098
   [junit4]   2> 17532 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 17532 T17 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 17532 T17 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541
   [junit4]   2> 17532 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\'
   [junit4]   2> 17567 T17 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\solr.xml
   [junit4]   2> 17670 T17 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 17675 T17 oasc.CoreContainer.<init> New CoreContainer 9336124
   [junit4]   2> 17675 T17 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\]
   [junit4]   2> 17732 T17 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 17733 T17 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 17733 T17 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 17734 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 17734 T17 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 17735 T17 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 17736 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 17736 T17 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 17737 T17 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 17742 T17 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 17742 T17 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 17743 T17 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51091/solr
   [junit4]   2> 17765 T17 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 17767 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17779 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c2e163 name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17780 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17795 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17799 T40 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca8f43 name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 17800 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 17812 T17 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 17833 T17 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 17854 T17 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 17861 T17 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51098_
   [junit4]   2> 17868 T17 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51098_
   [junit4]   2> 17881 T17 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 17913 T17 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 17922 T17 oasc.Overseer.start Overseer (id=90640112341680131-127.0.0.1:51098_-n_0000000000) starting
   [junit4]   2> 17948 T17 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 17971 T17 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 17973 T42 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 17983 T17 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 17990 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 18017 T41 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 18030 T17 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541
   [junit4]   2> 18277 T17 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\conf\
   [junit4]   2> 18290 T17 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 18290 T17 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 18291 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 18371 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 18376 T45 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff83a9 name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 18377 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 18381 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 18391 T17 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 18855 T17 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 18859 T17 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51110
   [junit4]   2> 18861 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 18861 T17 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 18862 T17 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648
   [junit4]   2> 18862 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648\'
   [junit4]   2> 18892 T17 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648\solr.xml
   [junit4]   2> 18987 T17 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 18988 T17 oasc.CoreContainer.<init> New CoreContainer 14820075
   [junit4]   2> 18988 T17 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648\]
   [junit4]   2> 18991 T17 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 18992 T17 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 18993 T17 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 18993 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 18994 T17 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 18994 T17 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 18994 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 18994 T17 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 18994 T17 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19001 T17 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19001 T17 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19002 T17 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51091/solr
   [junit4]   2> 19002 T17 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 19004 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19015 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62af46 name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19016 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19024 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19028 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4854f6 name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19028 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19043 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 20080 T17 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51110_
   [junit4]   2> 20084 T17 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51110_
   [junit4]   2> 20122 T40 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20156 T58 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20156 T45 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 20176 T17 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648
   [junit4]   2> 20416 T17 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383058357648\conf\
   [junit4]   2> 20430 T17 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 20431 T17 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 20432 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 20896 T17 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 20899 T17 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51119
   [junit4]   2> 20901 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 20901 T17 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 20901 T17 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693
   [junit4]   2> 20901 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\'
   [junit4]   2> 20932 T17 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\solr.xml
   [junit4]   2> 21022 T17 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 21023 T17 oasc.CoreContainer.<init> New CoreContainer 2107123
   [junit4]   2> 21023 T17 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\]
   [junit4]   2> 21025 T17 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 21026 T17 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 21026 T17 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21027 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21027 T17 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21028 T17 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21028 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21029 T17 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21029 T17 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21035 T17 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21035 T17 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21036 T17 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51091/solr
   [junit4]   2> 21036 T17 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 21040 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21052 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd20b6 name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21053 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21061 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21063 T19 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14204b3b2870007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 21068 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ad186f name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21069 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21082 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 22117 T17 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51119_
   [junit4]   2> 22120 T17 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51119_
   [junit4]   2> 22135 T45 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22136 T72 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22140 T40 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22140 T58 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 22156 T17 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693
   [junit4]   2> 22398 T17 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\conf\
   [junit4]   2> 22410 T17 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 22411 T17 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 22412 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 22886 T17 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 22890 T17 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51128
   [junit4]   2> 22892 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 22892 T17 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 22893 T17 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673
   [junit4]   2> 22893 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\'
   [junit4]   2> 22932 T17 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\solr.xml
   [junit4]   2> 23017 T17 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 23018 T17 oasc.CoreContainer.<init> New CoreContainer 10110193
   [junit4]   2> 23019 T17 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\]
   [junit4]   2> 23022 T17 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 23022 T17 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 23022 T17 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 23023 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 23023 T17 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 23024 T17 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 23024 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 23025 T17 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 23025 T17 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 23030 T17 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 23030 T17 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 23031 T17 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51091/solr
   [junit4]   2> 23031 T17 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 23033 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23043 T84 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@244f74 name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23044 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23052 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23057 T86 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d3633c name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23058 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23075 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24113 T17 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51128_
   [junit4]   2> 24116 T17 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51128_
   [junit4]   2> 24130 T58 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 24131 T45 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 24132 T40 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 24133 T72 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 24136 T86 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 24154 T17 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673
   [junit4]   2> 24392 T17 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\conf\
   [junit4]   2> 24405 T17 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 24406 T17 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 24406 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 24870 T17 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24874 T17 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51137
   [junit4]   2> 24874 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24874 T17 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24874 T17 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: .\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668
   [junit4]   2> 24876 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\'
   [junit4]   2> 24908 T17 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\solr.xml
   [junit4]   2> 24993 T17 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 24993 T17 oasc.CoreContainer.<init> New CoreContainer 8790081
   [junit4]   2> 24993 T17 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\]
   [junit4]   2> 24996 T17 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 24996 T17 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 24996 T17 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24996 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24996 T17 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24998 T17 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24998 T17 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24998 T17 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24998 T17 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25004 T17 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25004 T17 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25004 T17 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51091/solr
   [junit4]   2> 25004 T17 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 25007 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25038 T100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1af9d1c name:ZooKeeperConnection Watcher:127.0.0.1:51091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25039 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25048 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25052 T102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ede19e name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25053 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25068 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26105 T17 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51137_
   [junit4]   2> 26108 T17 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51137_
   [junit4]   2> 26134 T45 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26137 T58 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26137 T86 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26137 T40 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26140 T72 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26144 T102 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 26158 T17 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668
   [junit4]   2> 26401 T17 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\conf\
   [junit4]   2> 26414 T17 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 26416 T17 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 26417 T17 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26424 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26429 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb800f name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26430 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26433 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26455 T17 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26460 T107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1900f17 name:ZooKeeperConnection Watcher:127.0.0.1:51091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26460 T17 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26465 T17 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 27311 T36 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 27317 T40 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27328 T42 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 27334 T42 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 28735 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 28746 T41 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 28746 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 28791 T102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28791 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28791 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28803 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28804 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28805 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28807 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28811 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 28872 T42 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 28873 T42 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:51098_
   [junit4]   2> 28894 T42 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:51137_
   [junit4]   2> 28896 T42 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:51119_
   [junit4]   2> 28897 T42 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:51128_
   [junit4]   2> 28947 T94 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 28949 T35 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 28951 T63 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 28952 T78 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 28955 T94 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 28955 T35 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 28957 T63 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 28958 T78 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30212 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30214 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51137",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51137_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 30237 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51098",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51098_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 30251 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51119",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51119_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 30285 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51128",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51128_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 30303 T102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30303 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30304 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30306 T72 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30307 T86 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30315 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30317 T107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30318 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30800 T94 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 30802 T35 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 30804 T63 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 30807 T78 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 30811 T94 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: .\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2
   [junit4]   2> 30811 T35 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1
   [junit4]   2> 30812 T35 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 30812 T94 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 30812 T78 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2
   [junit4]   2> 30814 T78 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 30815 T35 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 30815 T35 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 30827 T94 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 30827 T94 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 30829 T35 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 30829 T35 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 30832 T78 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 30832 T78 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 30836 T63 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1
   [junit4]   2> 30836 T63 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 30841 T94 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 30841 T94 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 30841 T78 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 30842 T78 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 30844 T63 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 30845 T63 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 30849 T35 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 30853 T63 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 30857 T35 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\'
   [junit4]   2> 30860 T78 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 30864 T94 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 30876 T78 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\'
   [junit4]   2> 30908 T94 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\'
   [junit4]   2> 30912 T63 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 30926 T63 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\'
   [junit4]   2> 31030 T35 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 31068 T78 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 31186 T78 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 31188 T35 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 31195 T94 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 31263 T35 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 31282 T94 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 31291 T63 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 31309 T78 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 31388 T63 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 31409 T35 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 31465 T78 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 31477 T94 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 31533 T94 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 31589 T63 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 32062 T63 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 34522 T63 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34524 T63 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34563 T78 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34568 T78 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34574 T63 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34621 T78 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34627 T63 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34634 T63 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34666 T78 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34752 T78 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34766 T35 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34766 T35 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34770 T35 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34777 T94 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34777 T94 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 34781 T94 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34787 T35 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34795 T35 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34869 T94 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34882 T94 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 35051 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35056 T35 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35072 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35084 T78 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35094 T63 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35099 T94 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35106 T78 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35133 T35 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35140 T94 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35140 T94 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35140 T94 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35146 T78 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35158 T63 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 35160 T94 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35160 T94 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35160 T94 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 35162 T94 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at .\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\, dataDir=.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\data\
   [junit4]   2> 35170 T94 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@935741
   [junit4]   2> 35177 T94 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\data\
   [junit4]   2> 35177 T94 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\data\index/
   [junit4]   2> 35177 T94 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '.\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\data\index' doesn't exist. Creating new index...
   [junit4]   2> 35177 T94 oasc.CachingDirectoryFactory.get return new directory for .\org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383058363668\delLiveColl_shard1_replica2\data\index
   [junit4]   2> 35178 T94 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=21.8251953125, floorSegmentMB=0.943359375, forceMergeDeletesPctAllowed=10.612121086341341, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 35179 T94 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@e86f41 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@81d783),segFN=segments_1,generation=1}
   [junit4]   2> 35185 T94 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35190 T78 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35191 T78 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35192 T78 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35198 T35 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35200 T63 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35203 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 35204 T78 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35205 T78 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35205 T78 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 35209 T35 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35210 T35 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35209 T78 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\, dataDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\data\
   [junit4]   2> 35210 T35 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35213 T78 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@935741
   [junit4]   2> 35212 T63 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35220 T63 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35220 T63 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 35242 T63 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35242 T63 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35242 T63 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 35245 T35 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 35245 T35 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 35245 T35 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 35248 T35 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\, dataDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\data\
   [junit4]   2> 35248 T35 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@935741
   [junit4]   2> 35251 T63 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\, dataDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\data\
   [junit4]   2> 35251 T63 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@935741
   [junit4]   2> 35260 T35 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\data\
   [junit4]   2> 35260 T35 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\data\index/
   [junit4]   2> 35260 T35 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 35261 T63 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\data\
   [junit4]   2> 35261 T63 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\data\index/
   [junit4]   2> 35262 T63 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 35263 T35 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383058355541\delLiveColl_shard1_replica1\data\index
   [junit4]   2> 35263 T63 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383058359693\delLiveColl_shard2_replica1\data\index
   [junit4]   2> 35264 T78 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\data\
   [junit4]   2> 35264 T78 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\data\index/
   [junit4]   2> 35264 T78 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\data\index' doesn't exist. Creating new index...
   [junit4]   2> 35264 T78 oasc.CachingDirectoryFactory.get return new directory for ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build\solr-core\test\J0\.\org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383058361673\delLiveColl_shard2_replica2\data\index
   [junit4]   2> 35268 T78 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=21.8251953125, floorSegmentMB=0.943359375, forceMergeDeletesPctAllowed=10.612121086341341, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 35266 T63 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=21.8251953125, floorSegmentMB=0.943359375, forceMergeDeletesPctAllowed=10.612121086341341, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 35270 T35 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=21.8251953125, floorSegmentMB=0.943359375, forceMergeDeletesPctAllowed=10.612121086341341, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 35273 T63 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@129552 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1454f68),segFN=segments_1,generation=1}
   [junit4]   2> 35273 T63 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35281 T78 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@128cad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@525531),segFN=segments_1,generation=1}
   [junit4]   2> 35281 T78 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35289 T35 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@127a20c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@270664),segFN=segments_1,generation=1}
   [junit4]   2> 35289 T35 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 35300 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 35306 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 35306 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35307 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35307 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 35308 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 35309 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 35310 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 35311 T78 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 35311 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35312 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35312 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 35337 T78 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 35351 T78 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 35351 T78 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35352 T78 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35352 T78 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 35406 T35 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 35417 T35 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 35430 T35 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 35430 T35 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35431 T35 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 35433 T35 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 35451 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35451 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35451 T63 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 35467 T63 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 35467 T78 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35469 T78 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35474 T78 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 35474 T35 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35477 T35 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35477 T35 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 35483 T78 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 35481 T35 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 35489 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35491 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 35491 T94 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 35513 T35 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 35514 T35 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 35515 T78 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 35515 T78 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 35520 T35 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 35521 T35 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 35523 T78 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 35523 T94 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 35529 T35 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 35530 T63 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 35530 T63 oasc.Reques

[...truncated too long message...]

a:69)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 245567 T16 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=107, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	   2) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=RUNNABLE, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 260829 T107 oaz.ClientCnxn$EventThread.run ERROR Event thread exiting due to interruption java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 
   [junit4]   2> 261078 T106 oaz.ClientCnxn$SendThread.run WARN Session 0x14204b3b287000e for server null, unexpected error, closing socket connection and attempting reconnect java.nio.channels.ClosedByInterruptException
   [junit4]   2> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:635)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 261330 T106 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 261830 T106 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 262740 T106 oaz.ClientCnxn$SendThread.run WARN Session 0x14204b3b287000e for server null, unexpected error, closing socket connection and attempting reconnect java.nio.channels.ClosedByInterruptException
   [junit4]   2> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:635)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 262831 T106 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 263740 T106 oaz.ClientCnxn$SendThread.run WARN Session 0x14204b3b287000e for server null, unexpected error, closing socket connection and attempting reconnect java.nio.channels.ClosedByInterruptException
   [junit4]   2> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:635)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 248571 T16 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ca, timezone=Pacific/Gambier
   [junit4]   2> NOTE: Windows 7 6.1 x86/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=2,threads=3,free=34203840,total=65470464
   [junit4]   2> NOTE: All tests run in this JVM: [DocValuesTest, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=44BEC085590C18F4 -Dtests.slow=true -Dtests.locale=ca -Dtests.timezone=Pacific/Gambier -Dtests.file.encoding=Cp1252
   [junit4] ERROR   0.00s | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:662)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=107, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=RUNNABLE, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255)
   [junit4]    >         at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
   [junit4]    >         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=106, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[44BEC085590C18F4]-SendThread(127.0.0.1:51091), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([44BEC085590C18F4]:0)
   [junit4] Completed in 248.82s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 994 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:428: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:408: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\build.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\solr\common-build.xml:483: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:1263: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-4.x-Windows\lucene\common-build.xml:906: There were test failures: 330 suites, 1457 tests, 3 suite-level errors, 1 error, 1366 ignored

Total time: 33 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure