You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/05 22:23:02 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 36008 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:137: Source checkout is dirty after running tests!!! Offending files:
* ./lucene/.test-totals-922324336062316929.tmp
* ./solr/.test-totals-5710645736889619892.tmp

Total time: 50 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8193 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8193/
Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node3 in shard/collection : shard1/delLiveColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node3 in shard/collection : shard1/delLiveColl
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966:720CA955DC83F95A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:149)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:135)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)




Build Log:
[...truncated 10744 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1316998 T3221 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1317001 T3221 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-DeleteReplicaTest-1383706308253
   [junit4]   2> 1317002 T3221 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1317003 T3222 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1317103 T3221 oasc.ZkTestServer.run start zk server on port:53963
   [junit4]   2> 1317114 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317198 T3228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bf416ae name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317198 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317198 T3221 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1317203 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317204 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67cfe1c5 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317205 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317205 T3221 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1317208 T3221 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1317210 T3221 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1317218 T3221 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1317224 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1317225 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1317228 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1317229 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1317231 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1317232 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1317234 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1317234 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1317237 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1317237 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1317239 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1317240 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1317242 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1317242 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1317245 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1317245 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1317247 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1317247 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1317250 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1317250 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1317252 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1317252 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1317315 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1317319 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46383
   [junit4]   2> 1317320 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/'
   [junit4]   2> 1317329 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/solr.xml
   [junit4]   2> 1317358 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1317358 T3221 oasc.CoreContainer.<init> New CoreContainer 1178080130
   [junit4]   2> 1317358 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/]
   [junit4]   2> 1317359 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1317362 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1317362 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1317362 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1317363 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1317363 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317366 T3241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a0b2d2 name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317366 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317368 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317370 T3243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b4c53af name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317370 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317372 T3221 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1317375 T3221 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1317377 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1317378 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46383_
   [junit4]   2> 1317379 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46383_
   [junit4]   2> 1317381 T3221 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1317385 T3221 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1317387 T3221 oasc.Overseer.start Overseer (id=90682576617865219-127.0.0.1:46383_-n_0000000000) starting
   [junit4]   2> 1317389 T3221 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1317392 T3245 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1317392 T3221 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1317402 T3221 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1317405 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1317407 T3244 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1317409 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506
   [junit4]   2> 1317409 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/conf/
   [junit4]   2> 1317438 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1317438 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1317439 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1317440 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317442 T3248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6516790d name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317442 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317456 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1317458 T3221 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1317517 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1317518 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52279
   [junit4]   2> 1317519 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/'
   [junit4]   2> 1317529 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/solr.xml
   [junit4]   2> 1317561 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1317562 T3221 oasc.CoreContainer.<init> New CoreContainer 1561880160
   [junit4]   2> 1317562 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/]
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1317565 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1317565 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1317566 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1317566 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1317566 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1317567 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1317567 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317569 T3259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@436f2ac1 name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317569 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317572 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317573 T3261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42d67075 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317573 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317578 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1318581 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52279_
   [junit4]   2> 1318582 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52279_
   [junit4]   2> 1318585 T3243 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318585 T3261 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318585 T3248 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318590 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709
   [junit4]   2> 1318592 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/conf/
   [junit4]   2> 1318619 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1318620 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1318620 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1318679 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1318681 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53439
   [junit4]   2> 1318681 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/'
   [junit4]   2> 1318691 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/solr.xml
   [junit4]   2> 1318730 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1318731 T3221 oasc.CoreContainer.<init> New CoreContainer 2101110435
   [junit4]   2> 1318731 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/]
   [junit4]   2> 1318732 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1318732 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1318735 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1318736 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1318736 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1318737 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1318737 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1318738 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1318739 T3273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a4e869e name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1318740 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1318742 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1318743 T3275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d1c058 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1318744 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1318747 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1319749 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53439_
   [junit4]   2> 1319751 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53439_
   [junit4]   2> 1319754 T3261 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319754 T3248 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319754 T3275 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319755 T3243 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319758 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872
   [junit4]   2> 1319760 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/conf/
   [junit4]   2> 1319786 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1319787 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1319787 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1319844 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1319846 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35150
   [junit4]   2> 1319847 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/'
   [junit4]   2> 1319856 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/solr.xml
   [junit4]   2> 1319883 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1319884 T3221 oasc.CoreContainer.<init> New CoreContainer 1596363683
   [junit4]   2> 1319884 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/]
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1319887 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1319888 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1319888 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1319888 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1319889 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1319890 T3287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44f218a name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1319890 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1319893 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1319895 T3289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f3776cd name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1319895 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1319899 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1320901 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35150_
   [junit4]   2> 1320902 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35150_
   [junit4]   2> 1320905 T3261 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3248 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320905 T3275 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3289 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3243 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320910 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039
   [junit4]   2> 1320911 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/conf/
   [junit4]   2> 1320931 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1320931 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1320932 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1320995 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1320998 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36332
   [junit4]   2> 1320999 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1320999 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1321000 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184
   [junit4]   2> 1321000 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/'
   [junit4]   2> 1321010 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/solr.xml
   [junit4]   2> 1321038 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1321038 T3221 oasc.CoreContainer.<init> New CoreContainer 695590024
   [junit4]   2> 1321039 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/]
   [junit4]   2> 1321039 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1321042 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1321042 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1321043 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1321043 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1321044 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1321045 T3301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199f332c name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1321045 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1321048 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1321049 T3303 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ef8bd2 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1321049 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1321052 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322055 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36332_
   [junit4]   2> 1322056 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36332_
   [junit4]   2> 1322059 T3248 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3261 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322060 T3243 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3303 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3275 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322060 T3289 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322064 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184
   [junit4]   2> 1322065 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/conf/
   [junit4]   2> 1322093 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1322093 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1322094 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1322100 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1322102 T3306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9f73ad name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1322103 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1322104 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322113 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1322114 T3308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@663cec84 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1322114 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1322115 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322118 T3236 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 1322120 T3243 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1322121 T3245 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1322121 T3245 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1323418 T3244 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1323419 T3244 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1323419 T3244 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1323423 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323526 T3245 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1323526 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:35150_
   [junit4]   2> 1323527 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:46383_
   [junit4]   2> 1323527 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:36332_
   [junit4]   2> 1323528 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:52279_
   [junit4]   2> 1323530 T3284 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 1323530 T3255 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 1323530 T3238 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 1323531 T3298 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 1323532 T3284 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3238 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3298 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3255 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1324939 T3244 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1324939 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35150",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35150_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324944 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52279",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52279_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324946 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46383",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46383_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324949 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36332",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36332_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324996 T3308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1325532 T3284 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1325532 T3238 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1325533 T3255 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1325533 T3284 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1
   [junit4]   2> 1325533 T3298 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1325534 T3255 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2
   [junit4]   2> 1325533 T3284 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325533 T3238 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2
   [junit4]   2> 1325534 T3238 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325534 T3298 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1
   [junit4]   2> 1325534 T3255 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325535 T3238 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325535 T3284 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325535 T3298 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325536 T3284 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325536 T3255 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325537 T3255 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325537 T3284 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325536 T3238 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325537 T3284 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325538 T3255 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325538 T3255 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325536 T3298 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325539 T3298 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325539 T3238 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325540 T3238 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325541 T3284 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325541 T3298 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325542 T3255 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325542 T3238 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325543 T3284 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/'
   [junit4]   2> 1325544 T3298 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325544 T3238 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/'
   [junit4]   2> 1325545 T3298 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/'
   [junit4]   2> 1325546 T3255 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/'
   [junit4]   2> 1325599 T3284 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325600 T3298 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325604 T3238 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325621 T3255 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325644 T3284 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325645 T3298 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325645 T3284 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325646 T3298 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325652 T3298 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1325652 T3284 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1325664 T3238 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325666 T3238 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325677 T3238 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1325693 T3255 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325694 T3255 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325706 T3255 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1326169 T3238 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326175 T3238 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326175 T3298 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326177 T3238 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326180 T3298 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326183 T3298 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326190 T3238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326194 T3238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326197 T3284 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326198 T3238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326199 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326202 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326203 T3238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326204 T3284 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326207 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326208 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326208 T3238 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326208 T3284 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326208 T3238 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/
   [junit4]   2> 1326209 T3238 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326210 T3298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326210 T3238 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data
   [junit4]   2> 1326211 T3238 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 1326211 T3238 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326212 T3238 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 1326213 T3238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326214 T3238 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c31b185 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40dc859a),segFN=segments_1,generation=1}
   [junit4]   2> 1326214 T3238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326215 T3298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326223 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326223 T3238 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326223 T3298 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326224 T3238 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326224 T3238 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326225 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326226 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326226 T3298 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326228 T3238 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326229 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326229 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326230 T3298 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326230 T3298 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/
   [junit4]   2> 1326230 T3284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326230 T3298 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326232 T3298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data
   [junit4]   2> 1326232 T3298 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 1326232 T3298 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326233 T3238 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326233 T3298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 1326233 T3298 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326235 T3298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7a00106d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f31458d),segFN=segments_1,generation=1}
   [junit4]   2> 1326235 T3298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326236 T3284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326236 T3238 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326237 T3238 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326237 T3238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326238 T3238 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c31b185 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40dc859a),segFN=segments_1,generation=1}
   [junit4]   2> 1326238 T3238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326239 T3238 oass.SolrIndexSearcher.<init> Opening Searcher@382a68f0 main
   [junit4]   2> 1326239 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326240 T3284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326241 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326241 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326242 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326242 T3313 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@382a68f0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326242 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326243 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326243 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326243 T3298 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326244 T3284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326243 T3238 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 1326244 T3298 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326244 T3238 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:46383 collection:delLiveColl shard:shard1
   [junit4]   2> 1326245 T3298 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326245 T3298 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326245 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326246 T3238 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 1326246 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326246 T3298 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326246 T3284 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326247 T3298 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326247 T3284 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/
   [junit4]   2> 1326247 T3298 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326248 T3284 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326249 T3284 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data
   [junit4]   2> 1326249 T3298 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326249 T3284 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 1326250 T3284 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326256 T3284 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 1326257 T3284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326257 T3284 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a276ebf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8214ea8),segFN=segments_1,generation=1}
   [junit4]   2> 1326258 T3284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326258 T3298 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326258 T3298 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326259 T3298 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326259 T3298 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7a00106d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f31458d),segFN=segments_1,generation=1}
   [junit4]   2> 1326260 T3298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326260 T3298 oass.SolrIndexSearcher.<init> Opening Searcher@7a78aa9a main
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326261 T3238 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326262 T3314 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@7a78aa9a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326263 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326263 T3298 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 1326263 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326263 T3298 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:36332 collection:delLiveColl shard:shard2
   [junit4]   2> 1326264 T3238 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 1326264 T3284 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326264 T3284 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326265 T3298 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326266 T3284 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326266 T3284 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326267 T3284 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326270 T3284 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326270 T3284 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326270 T3298 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1326270 T3284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326271 T3284 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a276ebf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8214ea8),segFN=segments_1,generation=1}
   [junit4]   2> 1326271 T3284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326272 T3284 oass.SolrIndexSearcher.<init> Opening Searcher@6f609d41 main
   [junit4]   2> 1326272 T3298 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=180000
   [junit4]   2> 1326274 T3315 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@6f609d41 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326275 T3284 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 1326275 T3284 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:35150 collection:delLiveColl shard:shard1
   [junit4]   2> 1326288 T3255 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326295 T3255 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326298 T3255 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326311 T3255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326315 T3255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326319 T3255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326323 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326323 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326323 T3255 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326324 T3255 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/
   [junit4]   2> 1326324 T3255 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326325 T3255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data
   [junit4]   2> 1326325 T3255 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index/
   [junit4]   2> 1326326 T3255 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326326 T3255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index
   [junit4]   2> 1326327 T3255 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326328 T3255 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3a2f7124 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3eee2ac5),segFN=segments_1,generation=1}
   [junit4]   2> 1326328 T3255 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326332 T3255 oasup.UpdateRequestProcessorChain.init creating updateRequestProces

[...truncated too long message...]

 1419881 T3307 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1420381 T3307 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1420382 T3307 oaz.ClientCnxn$SendThread.run WARN Session 0x1422b52aea0000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 103884 T3220 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=zh_HK, timezone=America/Indiana/Tell_City
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=3,free=378498024,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestQueryTypes, TestSolr4Spatial, ShardSplitTest, PathHierarchyTokenizerFactoryTest, DirectSolrConnectionTest, LeaderElectionIntegrationTest, SpellCheckComponentTest, TestQuerySenderNoQuery, SimplePostToolTest, ZkNodePropsTest, TestIndexingPerformance, TestQuerySenderListener, SuggesterWFSTTest, TestCoreDiscovery, TestStressReorder, StandardRequestHandlerTest, BJQParserTest, TestArbitraryIndexDir, TestCSVResponseWriter, AddSchemaFieldsUpdateProcessorFactoryTest, TestWordDelimiterFilterFactory, TestComponentsName, TestSolrJ, TestIBSimilarityFactory, RequiredFieldsTest, TestShardHandlerFactory, TestDynamicFieldCollectionResource, TestRangeQuery, TestFastOutputStream, TestCloudManagedSchemaAddField, HdfsChaosMonkeySafeLeaderTest, SortByFunctionTest, TestDFRSimilarityFactory, TestLazyCores, BadIndexSchemaTest, TestRecoveryHdfs, SolrRequestParserTest, AlternateDirectoryTest, TestSchemaResource, PluginInfoTest, RAMDirectoryFactoryTest, ChaosMonkeySafeLeaderTest, DateFieldTest, ZkSolrClientTest, TestSweetSpotSimilarityFactory, DistanceFunctionTest, TestNumberUtils, TestIndexSearcher, PrimUtilsTest, TestAtomicUpdateErrorCases, TestDistribDocBasedVersion, BlockCacheTest, TestAnalyzedSuggestions, FileUtilsTest, DocumentAnalysisRequestHandlerTest, TestStressRecovery, OpenExchangeRatesOrgProviderTest, DeleteInactiveReplicaTest, JSONWriterTest, SolrCoreCheckLockOnStartupTest, ConvertedLegacyTest, CurrencyFieldOpenExchangeTest, TestCollapseQParserPlugin, TestCSVLoader, StressHdfsTest, StatsComponentTest, TestFiltering, CollectionsAPIDistributedZkTest, TestUpdate, SolrCmdDistributorTest, DirectSolrSpellCheckerTest, IndexSchemaRuntimeFieldTest, HdfsBasicDistributedZk2Test, CoreMergeIndexesAdminHandlerTest, SpellCheckCollatorTest, TestFuzzyAnalyzedSuggestions, FullSolrCloudDistribCmdsTest, TestDefaultSimilarityFactory, CachingDirectoryFactoryTest, UniqFieldsUpdateProcessorFactoryTest, BasicZkTest, TestZkChroot, TestRandomDVFaceting, TestDistributedSearch, TestFaceting, TestHashPartitioner, ZkControllerTest, TestJoin, PeerSyncTest, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, TestFoldingMultitermQuery, SuggesterTSTTest, NoCacheHeaderTest, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestWriterPerf, TestReversedWildcardFilterFactory, TestOmitPositions, RequestHandlersTest, TestValueSourceCache, MoreLikeThisHandlerTest, FastVectorHighlighterTest, TestJmxIntegration, ReturnFieldsTest, MBeansHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, SearchHandlerTest, HighlighterConfigTest, ResponseLogComponentTest, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, MinimalSchemaTest, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCharFilters, SynonymTokenizerTest, EchoParamsTest, TestLMDirichletSimilarityFactory, TimeZoneUtilsTest, TestSystemIdResolver, SpellingQueryConverterTest, DOMUtilTest, TestDocumentBuilder, TestRTGBase, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=F3EA274DABDC9966 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK -Dtests.timezone=America/Indiana/Tell_City -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
   [junit4] Completed on J1 in 103.94s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 3 suite-level errors, 1 error, 35 ignored (6 assumptions)

Total time: 44 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8192 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8192/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate

Error Message:
expected:<[in]active> but was:<[]active>

Stack Trace:
org.junit.ComparisonFailure: expected:<[in]active> but was:<[]active>
	at __randomizedtesting.SeedInfo.seed([4289FDB0D725E1A4:92EB70FC89F03B20]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate(SliceStateUpdateTest.java:183)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10062 lines...]
   [junit4] Suite: org.apache.solr.cloud.SliceStateUpdateTest
   [junit4]   2> 415330 T1199 oas.SolrTestCaseJ4.setUp ###Starting testSliceStateUpdate
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233
   [junit4]   2> 415330 T1199 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 415331 T1200 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 415431 T1199 oasc.ZkTestServer.run start zk server on port:51293
   [junit4]   2> 415432 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415520 T1206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4347a370 name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415520 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415520 T1199 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 415525 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415526 T1208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c438065 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415526 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415526 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 415531 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 415534 T1199 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 415536 T1199 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 415538 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 415538 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 415541 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 415542 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 415544 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 415544 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 415546 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 415546 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 415548 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 415548 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 415550 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 415550 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 415552 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 415553 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 415555 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 415555 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 415557 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 415557 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 415559 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 415559 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 415561 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 415562 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 415564 T1199 oasc.SliceStateUpdateTest.setUp ####SETUP_START testSliceStateUpdate
   [junit4]   2> 415565 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415566 T1210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70a2114 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415566 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415566 T1199 oascc.SolrZkClient.makePath makePath: /collections/testcore
   [junit4]   2> 415569 T1199 oascc.SolrZkClient.makePath makePath: /collections/testcore/shards
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 415584 T1199 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 415618 T1199 oasc.CoreContainer.<init> New CoreContainer 915571128
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 415628 T1199 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 415659 T1199 oasc.CoreContainer.<init> New CoreContainer 22158319
   [junit4]   2> 415659 T1199 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 415662 T1199 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 415662 T1199 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 415663 T1199 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51293/solr
   [junit4]   2> 415663 T1199 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 415664 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415665 T1212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6236bbb6 name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415665 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415668 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415669 T1214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1313a56f name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415669 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415671 T1199 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 415674 T1199 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 415676 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 415678 T1199 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 415679 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 415681 T1199 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 415684 T1199 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 415686 T1199 oasc.Overseer.start Overseer (id=90682159414312964-127.0.0.1:8983_solr-n_0000000000) starting
   [junit4]   2> 415691 T1199 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 415694 T1216 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 415695 T1199 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 415697 T1199 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 415698 T1199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 415700 T1215 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 415702 T1217 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 415702 T1217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 415703 T1217 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 417205 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 417206 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 417206 T1215 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 417212 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 417703 T1217 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 417704 T1217 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 417704 T1217 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 417705 T1217 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 417706 T1217 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 417708 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 417709 T1217 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 417709 T1217 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 417767 T1217 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 417811 T1217 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 417813 T1217 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 417819 T1217 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 418122 T1217 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 418127 T1217 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 418129 T1217 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 418148 T1217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 418152 T1217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 418156 T1217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 418160 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 418160 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 418160 T1217 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 418161 T1217 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/
   [junit4]   2> 418161 T1217 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30eb5ea0
   [junit4]   2> 418162 T1217 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 418162 T1217 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index/
   [junit4]   2> 418163 T1217 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index' doesn't exist. Creating new index...
   [junit4]   2> 418163 T1217 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 418164 T1217 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1997227149, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 418165 T1217 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7c4f7bc8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20704adf)),segFN=segments_1,generation=1}
   [junit4]   2> 418166 T1217 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 418166 T1217 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 418171 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 418172 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 418172 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 418173 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 418173 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 418174 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 418174 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 418175 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 418175 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 418176 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 418177 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 418177 T1217 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 418177 T1217 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 418178 T1217 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 418179 T1217 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 418180 T1217 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 418181 T1217 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 418183 T1217 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 418183 T1217 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 418184 T1217 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 418185 T1217 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 418185 T1217 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 418186 T1217 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 418187 T1217 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418188 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418189 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418190 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418191 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418192 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418192 T1217 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418193 T1217 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418194 T1217 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 418194 T1217 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 418196 T1217 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 418197 T1217 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 418198 T1217 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [ENUMSCONFIG.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML]
   [junit4]   2> 418200 T1217 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 418200 T1217 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 418201 T1217 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 418201 T1217 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7c4f7bc8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20704adf)),segFN=segments_1,generation=1}
   [junit4]   2> 418201 T1217 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 418201 T1217 oass.SolrIndexSearcher.<init> Opening Searcher@4826a22e main
   [junit4]   2> 418202 T1217 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 418205 T1217 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 418224 T1218 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4826a22e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 418227 T1217 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 418228 T1217 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:8983/solr collection:collection1 shard:shard1
   [junit4]   2> 418229 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 418234 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:8983/solr/collection1/ shard1
   [junit4]   2> 418236 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 418719 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 418729 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 418741 T1217 oasc.ZkController.register We are http://127.0.0.1:8983/solr/collection1/ and leader is http://127.0.0.1:8983/solr/collection1/
   [junit4]   2> 418742 T1217 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:8983/solr
   [junit4]   2> 418742 T1217 oasc.ZkController.checkRecovery WARN Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 418742 T1217 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 418742 T1217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 418744 T1217 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 418745 T1199 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 418746 T1199 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 418746 T1199 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 418748 T1199 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 418748 T1199 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 418748 T1199 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 418748 T1199 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 418749 T1199 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51293/solr
   [junit4]   2> 418749 T1199 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 418749 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418751 T1221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31ca775e name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418751 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418754 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418761 T1223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2398635e name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418761 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418765 T1199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 419768 T1199 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 419770 T1199 oasc.ZkController.createEphemeralLiveNode Found a previous node that still exists while trying to register a new live node /live_nodes/127.0.0.1:8983_solr - removing existing node to create another.
   [junit4]   2> 419770 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 419770 T1214 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 419770 T1223 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 419772 T1214 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 419773 T1223 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 419778 T1224 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 419778 T1224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 419779 T1224 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 419779 T1224 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 419779 T1224 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 419779 T1224 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 419780 T1224 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 419780 T1224 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 419819 T1224 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 419862 T1224 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 419864 T1224 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 419870 T1224 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 420170 T1224 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 420175 T1224 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 420177 T1224 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 420189 T1224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 420192 T1224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 420195 T1224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 420196 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 420196 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 420197 T1224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 420198 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 420198 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 420198 T1224 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 420198 T1224 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/
   [junit4]   2> 420198 T1224 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30eb5ea0
   [junit4]   2> 420199 T1224 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 420199 T1224 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index/
   [junit4]   2> 420199 T1224 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index' doesn't exist. Creating new index...
   [junit4]   2> 420199 T1224 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 420200 T1224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1997227149, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 420200 T1224 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@33d633b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75471931)),segFN=segments_1,generation=1}
   [junit4]   2> 420201 T1224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 420201 T1224 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 420206 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 420206 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 420206 T1224 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 420206 T1224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 420206 T1224 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 420207 T1224 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 420207 T1224 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 420210 T1224 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420210 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420212 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420212 T1224 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 420215 T1224 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 420217 T1224 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 420218 T1224 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [ENUMSCONFIG.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML]
   [junit4]   2> 420219 T1224 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 420220 T1224 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 420220 T1224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 420220 T1224 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@33d633b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75471931)),segFN=segments_1,generation=1}
   [junit4]   2> 420221 T1224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 420221 T1224 oass.SolrIndexSearcher.<init> Opening Searcher@b589bb5 main
   [junit4]   2> 420221 T1224 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 420222 T1224 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 420230 T1225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b589bb5 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 420236 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 420236 T1224 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 420236 T1224 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:8983/solr collection:collection1 shard:shard1
   [junit4]   2> 420237 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 420240 T1224 oasc.ZkController.register We are http://127.0.0.1:8983/solr/collection1/ and leader is http://127.0.0.1:8983/solr/collection1/
   [junit4]   2> 420240 T1224 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:8983/solr
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 420240 T1224 oasc.ZkController.checkRecovery WARN Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 420240 T1224 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 420240 T1224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 420242 T1224 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 420242 T1199 oasc.SliceStateUpdateTest.setUp ####SETUP_END testSliceStateUpdate
   [junit4]   2> 420243 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 420244 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 420245 T1228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b72bd6 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 420245 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 420246 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 420246 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 420249 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 470260 T1199 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=915571128
   [junit4]   2> 471335 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 471336 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 471336 T1215 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 471344 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 471344 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 473261 T1199 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 473262 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 473263 T1199 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6ffd9f54
   [junit4]   2> 473277 T1199 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 473277 T1199 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 473277 T1199 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 473277 T1199 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 473279 T1199 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2;done=false>>]
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 473280 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index;done=false>>]
   [junit4]   2> 473280 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 473281 T1215 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90682159414312964-127.0.0.1:8983_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 473283 T1199 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22158319
   [junit4]   2> 473284 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 473285 T1223 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:8983/solr/collection1/ shard1
   [junit4]   2> 473286 T1223 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 473295 T1223 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 473297 T1223 oasc.Overseer.start Overseer (id=90682159414312966-127.0.0.1:8983_solr-n_0000000001) starting
   [junit4]   2> 473300 T1230 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 473301 T1229 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 473302 T1229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473303 T1229 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 473311 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 474284 T1199 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 474284 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 474286 T1199 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6ace3348
   [junit4]   2> 474300 T1199 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 474300 T1199 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 474300 T1199 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 474301 T1199 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 474302 T1199 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2;done=false>>]
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 474304 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index;done=false>>]
   [junit4]   2> 474304 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 474306 T1229 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90682159414312966-127.0.0.1:8983_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 474310 T1199 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51293 51293
   [junit4]   2> 474606 T1199 oas.SolrTestCaseJ4.tearDown ###Ending testSliceStateUpdate
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SliceStateUpdateTest -Dtests.method=testSliceStateUpdate -Dtests.seed=4289FDB0D725E1A4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_LU -Dtests.timezone=Asia/Omsk -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 59.3s J0 | SliceStateUpdateTest.testSliceStateUpdate <<<
   [junit4]    > Throwable #1: org.junit.ComparisonFailure: expected:<[in]active> but was:<[]active>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4289FDB0D725E1A4:92EB70FC89F03B20]:0)
   [junit4]    > 	at org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate(SliceStateUpdateTest.java:183)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 474638 T1199 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 59317 T1198 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 475808 T1223 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=de_LU, timezone=Asia/Omsk
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=1,free=100096728,total=187760640
   [junit4]   2> NOTE: All tests run in this JVM: [TestMergePolicyConfig, TestCoreContainer, TestSolrXMLSerializer, TestSolrXmlPersistence, ResourceLoaderTest, TestConfig, TestSearchPerf, ParsingFieldUpdateProcessorsTest, QueryEqualityTest, PrimitiveFieldTypeTest, WordBreakSolrSpellCheckerTest, TestManagedSchema, TestLRUCache, BasicDistributedZkTest, TestLMDirichletSimilarityFactory, JsonLoaderTest, DirectUpdateHandlerOptimizeTest, UpdateRequestProcessorFactoryTest, DateMathParserTest, TestRealTimeGet, SyncSliceTest, UnloadDistributedZkTest, TestGroupingSearch, TestDocumentBuilder, TestFieldTypeCollectionResource, TestCharFilters, TestDocSet, LegacyHTMLStripCharFilterTest, TermsComponentTest, SynonymTokenizerTest, DocumentBuilderTest, TestDocBasedVersionConstraints, TermVectorComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestBM25SimilarityFactory, ZkControllerTest, DOMUtilTest, TestSystemIdResolver, AssignTest, TestReversedWildcardFilterFactory, SpatialFilterTest, TestFunctionQuery, TestSchemaSimilarityResource, SliceStateUpdateTest]
   [junit4] Completed on J0 in 60.59s, 1 test, 1 failure <<< FAILURES!

[...truncated 743 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 1 failure, 35 ignored (6 assumptions)

Total time: 46 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8191 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8191/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node1 in shard/collection : shard1/delLiveColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node1 in shard/collection : shard1/delLiveColl
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2:E36FA8B904214FDE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:149)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:135)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 9837 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 21331 T64 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 21340 T64 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DeleteReplicaTest-1383694728716
   [junit4]   2> 21343 T64 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 21346 T65 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 21445 T64 oasc.ZkTestServer.run start zk server on port:38544
   [junit4]   2> 21677 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21729 T66 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 23457 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68aa419e name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23458 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23460 T64 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 23513 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23514 T66 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1422aa1fe8a0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 23528 T73 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4926aaf7 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23528 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23535 T64 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 23544 T64 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 23550 T64 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 23577 T64 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 23607 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 23621 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 23635 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 23641 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 23665 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 23665 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 23673 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 23674 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 23696 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 23697 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 23701 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 23702 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 23723 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 23724 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 23736 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 23736 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 23751 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 23752 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 23766 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 23767 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 23774 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 23775 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 24120 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24346 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34437
   [junit4]   2> 24367 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24368 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24369 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174
   [junit4]   2> 24369 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/'
   [junit4]   2> 24388 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/solr.xml
   [junit4]   2> 24455 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 24458 T64 oasc.CoreContainer.<init> New CoreContainer 968317684
   [junit4]   2> 24459 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/]
   [junit4]   2> 24475 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 24476 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 24477 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24477 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24478 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24478 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24479 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24479 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24480 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24482 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24483 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24483 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 24495 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 24496 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24520 T85 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a97e32c name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24521 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24549 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24554 T87 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44ed040f name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24555 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24564 T64 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 24589 T64 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 24623 T64 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 24652 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34437_
   [junit4]   2> 24673 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34437_
   [junit4]   2> 24682 T64 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 24714 T64 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 24718 T64 oasc.Overseer.start Overseer (id=90681817749389315-127.0.0.1:34437_-n_0000000000) starting
   [junit4]   2> 24755 T64 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 24790 T89 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 24791 T64 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 24796 T64 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 24803 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24828 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174
   [junit4]   2> 24832 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/conf/
   [junit4]   2> 24839 T88 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 24873 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 24874 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 24874 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 24905 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24911 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f5e8edc name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24911 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24923 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24934 T64 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 25067 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 25070 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36637
   [junit4]   2> 25071 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 25072 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 25072 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310
   [junit4]   2> 25073 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/'
   [junit4]   2> 25112 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/solr.xml
   [junit4]   2> 25189 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 25190 T64 oasc.CoreContainer.<init> New CoreContainer 1489186216
   [junit4]   2> 25191 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/]
   [junit4]   2> 25193 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 25193 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 25194 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 25194 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 25195 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 25195 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 25196 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 25196 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 25197 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25199 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25200 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25200 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 25201 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 25204 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25206 T103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4648548e name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25207 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25238 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25266 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437d3a6a name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25267 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25274 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26285 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36637_
   [junit4]   2> 26287 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36637_
   [junit4]   2> 26293 T92 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26294 T87 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26294 T105 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26301 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310
   [junit4]   2> 26302 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/conf/
   [junit4]   2> 26322 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 26323 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 26323 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26397 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26399 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60628
   [junit4]   2> 26400 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26400 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26401 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699
   [junit4]   2> 26401 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/'
   [junit4]   2> 26411 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/solr.xml
   [junit4]   2> 26449 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 26449 T64 oasc.CoreContainer.<init> New CoreContainer 35487695
   [junit4]   2> 26450 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/]
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26455 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26455 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26456 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 26456 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 26457 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26459 T117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b72935a name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26460 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26465 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26468 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@521effe1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26468 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26475 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 27480 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60628_
   [junit4]   2> 27482 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60628_
   [junit4]   2> 27486 T105 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27486 T87 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27486 T119 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27487 T92 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27492 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699
   [junit4]   2> 27493 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/conf/
   [junit4]   2> 27519 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 27520 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 27520 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 27589 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 27592 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39004
   [junit4]   2> 27592 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/'
   [junit4]   2> 27603 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/solr.xml
   [junit4]   2> 27641 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 27642 T64 oasc.CoreContainer.<init> New CoreContainer 899097207
   [junit4]   2> 27642 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/]
   [junit4]   2> 27643 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 27643 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 27646 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 27647 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 27647 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 27648 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 27648 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 27649 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27652 T131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63d36b29 name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27653 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27657 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27659 T133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a102e69 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27659 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27665 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 28669 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39004_
   [junit4]   2> 28671 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39004_
   [junit4]   2> 28674 T87 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T119 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T92 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T133 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T105 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28681 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896
   [junit4]   2> 28682 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/conf/
   [junit4]   2> 28708 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 28709 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 28709 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 28780 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 28783 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54488
   [junit4]   2> 28784 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 28784 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 28785 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086
   [junit4]   2> 28785 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/'
   [junit4]   2> 28802 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/solr.xml
   [junit4]   2> 28856 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 28857 T64 oasc.CoreContainer.<init> New CoreContainer 1871905209
   [junit4]   2> 28857 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/]
   [junit4]   2> 28859 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 28860 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 28860 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 28861 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 28861 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 28862 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 28862 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 28863 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 28863 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 28865 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 28866 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 28867 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 28867 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 28868 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28871 T145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48521df3 name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28872 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28876 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28885 T147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e1b71d1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28885 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28892 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29897 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54488_
   [junit4]   2> 29898 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54488_
   [junit4]   2> 29902 T87 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29903 T119 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29903 T105 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29902 T92 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29902 T147 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29904 T133 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29909 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086
   [junit4]   2> 29910 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/conf/
   [junit4]   2> 29930 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 29931 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 29931 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 29933 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29935 T150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@466054a1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29935 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29937 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29943 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29945 T152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46e05be2 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29945 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29947 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 30145 T143 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 30149 T87 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30152 T89 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 30154 T89 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 30856 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30858 T88 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 30859 T88 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 30875 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T87 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30959 T89 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 30960 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:34437_
   [junit4]   2> 30964 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:54488_
   [junit4]   2> 30965 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:36637_
   [junit4]   2> 30965 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:39004_
   [junit4]   2> 30985 T82 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 30985 T98 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 30985 T140 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 30985 T127 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 30988 T82 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30989 T98 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30990 T140 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30990 T127 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 32385 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32387 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34437",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34437_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32395 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36637",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36637_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32401 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54488",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54488_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32407 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39004",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39004_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32414 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T87 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32989 T82 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 32990 T98 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 32991 T140 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 32991 T127 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 32994 T140 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2
   [junit4]   2> 32994 T98 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1
   [junit4]   2> 32995 T98 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32994 T82 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1
   [junit4]   2> 32994 T127 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2
   [junit4]   2> 32995 T82 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32994 T140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32996 T127 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32997 T82 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32996 T98 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32997 T82 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32997 T127 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32998 T127 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32997 T140 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32999 T140 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32998 T98 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 33000 T82 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33001 T140 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33001 T98 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33000 T127 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33002 T98 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33001 T140 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33001 T82 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33002 T127 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33007 T98 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T82 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T140 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T127 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33010 T98 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/'
   [junit4]   2> 33010 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/'
   [junit4]   2> 33011 T140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/'
   [junit4]   2> 33012 T127 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/'
   [junit4]   2> 33102 T127 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33104 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33135 T98 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33159 T140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33164 T82 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33167 T82 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33190 T82 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 33198 T127 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33200 T127 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33203 T98 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33206 T98 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33231 T127 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 33234 T140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33236 T98 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 33243 T140 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33274 T140 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 33823 T98 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33835 T98 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33839 T82 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33840 T127 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33847 T82 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33851 T127 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33852 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33858 T98 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33864 T127 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33905 T82 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33906 T127 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33911 T98 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33911 T82 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33912 T127 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33918 T127 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33918 T98 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33920 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33934 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33935 T127 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33931 T82 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33935 T98 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33944 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33944 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33945 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33944 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33946 T82 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33945 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33947 T98 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33946 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33949 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33950 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33949 T127 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33951 T127 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/
   [junit4]   2> 33950 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33955 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33956 T98 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33956 T98 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/
   [junit4]   2> 33957 T98 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33955 T127 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33950 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33960 T82 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/
   [junit4]   2> 33960 T82 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33971 T98 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data
   [junit4]   2> 33972 T98 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 33972 T98 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33973 T98 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 33974 T98 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33976 T98 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@172be805 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71303650),segFN=segments_1,generation=1}
   [junit4]   2> 33976 T98 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 33979 T127 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data
   [junit4]   2> 33980 T127 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index/
   [junit4]   2> 33980 T127 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33981 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data
   [junit4]   2> 33981 T127 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index
   [junit4]   2> 33981 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 33982 T127 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33983 T82 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33984 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 33985 T82 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33986 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@143f0608 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@607dcebf),segFN=segments_1,generation=1}
   [junit4]   2> 33986 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 33995 T127 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@399cfbc4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@538f1db2),segFN=segments_1,generation=1}
   [junit4]   2> 33998 T127 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34003 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34004 T140 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34008 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34012 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34015 T140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34017 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34017 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34018 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34018 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34018 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34019 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34021 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34020 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34022 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34021 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34023 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34023 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34024 T127 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34022 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34027 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34023 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34027 T127 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34029 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T82 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34030 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T127 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34032 T127 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34033 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34032 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34033 T127 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34035 T98 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34037 T127 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34038 T127 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34038 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34038 T98 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34039 T82 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34041 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34041 T98 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34041 T127 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34041 T98 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34042 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34043 T98 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34043 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34045 T98 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34045 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34046 T98 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34049 T98 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34055 T127 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34055 T98 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34056 T98 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34056 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34056 T127 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34057 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34058 T98 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34058 T140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34059 T127 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34059 T98 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@172be805 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71303650),segFN=segments_1,generation=1}
   [junit4]   2> 34059 T82 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34060 T127 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@399cfbc4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@538f1db2),segFN=segments_1,generation=1}
   [junit4]   2> 34060 T98 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34061 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@143f0608 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@607dcebf),segFN=segments_1,generation=1}
   [junit4]   2> 34061 T127 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34062 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34062 T98 oass.SolrIndexSearcher.<init> Opening Searcher@7de39b29 main
   [junit4]   2> 34063 T82 oass.SolrIndexSearcher.<init> Opening Searcher@a5792d9 main
   [junit4]   2> 34063 T127 oass.SolrIndexSearcher.<init> Opening Searcher@6941817d main
   [junit4]   2> 34074 T140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34080 T140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 34084 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 34084 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 34085 T140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 34087 T158 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@7de39b29 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34091 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 34091 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 34093 T140 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 34094 T140 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/
   [junit4]   2> 34102 T140 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 34099 T157 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica2] Registered new searcher Searcher@6941817d main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34105 T159 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@a5792d9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34106 T98 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 34107 T98 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:36637 collection:delLiveColl shard:shard2
   [junit4]   2> 34116 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data
   [junit4]   2> 34117 T140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 34117 T140 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 34118 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 34118 T82 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 34119 T140 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 34120 T127 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica2
   [junit4]   2> 34119 T82 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:34437 collection:delLiveColl shard:shard1
   [junit4]   2> 34120 T127 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica2 address:http://127.0.0.1:39004 collection:delLiveColl shard:shard2
   [junit4]   2> 34121 T140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c207114 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2a1cf0bb),segFN=segments_1,generation=1}
   [junit4]   2> 34122 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34122 T82 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 34123 T127 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 34132 T98 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 34141 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34143 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34144 T127 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 34145 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34145 T82 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 34147 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34149 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34149 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34150 T82 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
   [junit4]   2> 34150 T127 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 34152 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 34152 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34153 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34153 T140 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34154 T127 oasc.SyncStrategy.sync Sync replica

[...truncated too long message...]

t(AbstractQueuedSynchronizer.java:2052)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 
   [junit4]   2> 129206 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 129207 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 129706 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 129707 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 130207 T151 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 130707 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 130708 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 131207 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 131208 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 110376 T63 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=pt, timezone=America/Detroit
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=2,free=84816760,total=126681088
   [junit4]   2> NOTE: All tests run in this JVM: [TermVectorComponentTest, CircularListTest, SolrCoreTest, UpdateParamsTest, TestMaxScoreQueryParser, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=628926A1737E2FE2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt -Dtests.timezone=America/Detroit -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
   [junit4] Completed on J0 in 110.46s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 955 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 3 suite-level errors, 1 error, 1075 ignored (1 assumption)

Total time: 34 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Thanks Uwe, will revert it and refine the patch. I am still not happy
with how this is implemented but I honestly don't think there is a
cleaner way to run such a "wrapper" target at the top-level of Ant
invocation (including sub-module level).

Dawid

On Tue, Nov 5, 2013 at 11:18 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> it looks like Dawid's commit placed these .tests.totals files in the wrong directory. Should be inside build! Maybe some property is incorrectly initialized.
>
> In addition I will add  another comment to the corresponding issue, because of some dependency of the commit to javascript, that’s not fully guaranteed to be available in every JVM (e.g., on FreeBSD we don't have it).
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
>> Sent: Tuesday, November 05, 2013 10:23 PM
>> To: dev@lucene.apache.org; erick@apache.org; dweiss@apache.org
>> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build #
>> 8189 - Still Failing!
>>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
>> Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 36008 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-
>> targets.xml:137: Source checkout is dirty after running tests!!! Offending
>> files:
>> * ./lucene/.test-totals-922324336062316929.tmp
>> * ./solr/.test-totals-5710645736889619892.tmp
>>
>> Total time: 50 minutes 15 seconds
>> Build step 'Invoke Ant' marked build as failure Description set: Java:
>> 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
>> Archiving artifacts Recording test results Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

it looks like Dawid's commit placed these .tests.totals files in the wrong directory. Should be inside build! Maybe some property is incorrectly initialized.

In addition I will add  another comment to the corresponding issue, because of some dependency of the commit to javascript, that’s not fully guaranteed to be available in every JVM (e.g., on FreeBSD we don't have it).

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Tuesday, November 05, 2013 10:23 PM
> To: dev@lucene.apache.org; erick@apache.org; dweiss@apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build #
> 8189 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
> Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 36008 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-
> targets.xml:137: Source checkout is dirty after running tests!!! Offending
> files:
> * ./lucene/.test-totals-922324336062316929.tmp
> * ./solr/.test-totals-5710645736889619892.tmp
> 
> Total time: 50 minutes 15 seconds
> Build step 'Invoke Ant' marked build as failure Description set: Java:
> 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
> Archiving artifacts Recording test results Email was triggered for: Failure
> Sending email for trigger: Failure
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org