You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/05 18:56:51 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8187 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8187/
Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)    2) Thread[id=1135, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   2) Thread[id=1135, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node1 in shard/collection : shard1/delLiveColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node1 in shard/collection : shard1/delLiveColl
	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D:9D67B3F55FEF2051]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:149)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:135)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10196 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 448057 T1048 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /j_mm/a
   [junit4]   2> 448061 T1048 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-DeleteReplicaTest-1383673999541
   [junit4]   2> 448061 T1048 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 448062 T1049 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 448162 T1048 oasc.ZkTestServer.run start zk server on port:39604
   [junit4]   2> 448163 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448240 T1055 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@187159 name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448240 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448240 T1048 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 448245 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448247 T1057 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cac5f1 name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448247 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448247 T1048 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 448250 T1048 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 448253 T1048 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 448255 T1048 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 448257 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 448257 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 448261 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 448262 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 448265 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 448265 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 448268 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 448268 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 448271 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 448272 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 448274 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 448275 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 448277 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 448277 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 448279 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 448280 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 448282 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 448282 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 448284 T1048 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 448285 T1048 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 448360 T1048 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 448363 T1048 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52277
   [junit4]   2> 448364 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 448364 T1048 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 448364 T1048 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767
   [junit4]   2> 448365 T1048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/'
   [junit4]   2> 448381 T1048 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/solr.xml
   [junit4]   2> 448421 T1048 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 448421 T1048 oasc.CoreContainer.<init> New CoreContainer 17721674
   [junit4]   2> 448422 T1048 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/]
   [junit4]   2> 448423 T1048 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 448423 T1048 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 448423 T1048 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 448423 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 448424 T1048 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 448424 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 448424 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 448424 T1048 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 448425 T1048 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 448426 T1048 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 448426 T1048 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 448426 T1048 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39604/solr
   [junit4]   2> 448427 T1048 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 448428 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448430 T1068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fa281d name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448431 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448433 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448434 T1070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28907d name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448435 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448437 T1048 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 448442 T1048 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 448445 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 448447 T1048 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52277_j_mm%2Fa
   [junit4]   2> 448448 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52277_j_mm%2Fa
   [junit4]   2> 448451 T1048 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 448455 T1048 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 448457 T1048 oasc.Overseer.start Overseer (id=90680459234050051-127.0.0.1:52277_j_mm%2Fa-n_0000000000) starting
   [junit4]   2> 448460 T1048 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 448465 T1072 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 448466 T1048 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 448468 T1048 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 448470 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448473 T1071 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 448474 T1048 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767
   [junit4]   2> 448475 T1048 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/conf/
   [junit4]   2> 448504 T1048 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 448504 T1048 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 448504 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 448506 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448508 T1075 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d4bb name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448508 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448511 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448514 T1048 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 448576 T1048 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 448583 T1048 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39878
   [junit4]   2> 448584 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 448584 T1048 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 448584 T1048 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994
   [junit4]   2> 448585 T1048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/'
   [junit4]   2> 448595 T1048 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/solr.xml
   [junit4]   2> 448628 T1048 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 448628 T1048 oasc.CoreContainer.<init> New CoreContainer 3480867
   [junit4]   2> 448629 T1048 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/]
   [junit4]   2> 448630 T1048 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 448630 T1048 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 448630 T1048 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 448631 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 448631 T1048 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 448632 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 448632 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 448632 T1048 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 448633 T1048 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 448634 T1048 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 448634 T1048 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 448635 T1048 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39604/solr
   [junit4]   2> 448635 T1048 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 448636 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448637 T1086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b64611 name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448638 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448641 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 448642 T1088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17791a5 name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 448642 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 448645 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 449648 T1048 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39878_j_mm%2Fa
   [junit4]   2> 449649 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39878_j_mm%2Fa
   [junit4]   2> 449653 T1088 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 449653 T1070 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 449653 T1075 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 449658 T1048 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994
   [junit4]   2> 449659 T1048 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/conf/
   [junit4]   2> 449684 T1048 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 449685 T1048 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 449685 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 449748 T1048 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 449750 T1048 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50024
   [junit4]   2> 449750 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 449751 T1048 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 449751 T1048 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166
   [junit4]   2> 449751 T1048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/'
   [junit4]   2> 449761 T1048 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/solr.xml
   [junit4]   2> 449819 T1048 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 449820 T1048 oasc.CoreContainer.<init> New CoreContainer 18084581
   [junit4]   2> 449820 T1048 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/]
   [junit4]   2> 449821 T1048 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 449821 T1048 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 449821 T1048 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 449821 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 449822 T1048 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 449822 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 449822 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 449823 T1048 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 449823 T1048 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 449824 T1048 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 449824 T1048 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 449824 T1048 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39604/solr
   [junit4]   2> 449825 T1048 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 449826 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449828 T1100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7de58e name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449828 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449832 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449833 T1102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71afd6 name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449833 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449837 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 450840 T1048 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50024_j_mm%2Fa
   [junit4]   2> 450841 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50024_j_mm%2Fa
   [junit4]   2> 450844 T1088 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 450845 T1070 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 450845 T1102 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 450845 T1075 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 450862 T1048 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166
   [junit4]   2> 450863 T1048 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/conf/
   [junit4]   2> 450882 T1048 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 450883 T1048 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 450883 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450947 T1048 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 450949 T1048 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60103
   [junit4]   2> 450949 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 450950 T1048 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 450950 T1048 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364
   [junit4]   2> 450950 T1048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/'
   [junit4]   2> 450960 T1048 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/solr.xml
   [junit4]   2> 450994 T1048 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 450995 T1048 oasc.CoreContainer.<init> New CoreContainer 9221151
   [junit4]   2> 450995 T1048 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/]
   [junit4]   2> 450996 T1048 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 450996 T1048 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 450996 T1048 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 450997 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 450997 T1048 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 450997 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 450998 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 450998 T1048 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 450998 T1048 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 450999 T1048 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 450999 T1048 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 451000 T1048 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39604/solr
   [junit4]   2> 451000 T1048 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 451001 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451003 T1114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77cd1d name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451003 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451008 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451009 T1116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b174d6 name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451009 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451012 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 452021 T1048 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60103_j_mm%2Fa
   [junit4]   2> 452022 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60103_j_mm%2Fa
   [junit4]   2> 452025 T1070 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 452025 T1075 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 452025 T1088 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 452025 T1102 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 452025 T1116 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 452030 T1048 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364
   [junit4]   2> 452031 T1048 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/conf/
   [junit4]   2> 452050 T1048 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 452051 T1048 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 452051 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 452143 T1048 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 452146 T1048 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40589
   [junit4]   2> 452146 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 452146 T1048 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 452147 T1048 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532
   [junit4]   2> 452147 T1048 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532/'
   [junit4]   2> 452157 T1048 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532/solr.xml
   [junit4]   2> 452194 T1048 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 452194 T1048 oasc.CoreContainer.<init> New CoreContainer 30285585
   [junit4]   2> 452194 T1048 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532/]
   [junit4]   2> 452195 T1048 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 452196 T1048 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 452196 T1048 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 452196 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 452196 T1048 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 452197 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 452197 T1048 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 452197 T1048 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 452197 T1048 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 452198 T1048 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 452199 T1048 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 452199 T1048 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39604/solr
   [junit4]   2> 452199 T1048 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 452200 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 452201 T1128 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c4b75 name:ZooKeeperConnection Watcher:127.0.0.1:39604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 452202 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 452205 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 452206 T1130 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17939e9 name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 452206 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 452212 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 453215 T1048 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40589_j_mm%2Fa
   [junit4]   2> 453216 T1048 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40589_j_mm%2Fa
   [junit4]   2> 453220 T1102 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453220 T1075 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453221 T1070 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453221 T1116 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453221 T1088 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453221 T1130 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 453225 T1048 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532
   [junit4]   2> 453226 T1048 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383674003532/conf/
   [junit4]   2> 453246 T1048 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 453246 T1048 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 453247 T1048 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 453248 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453250 T1133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ada30d name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453250 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453251 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 453255 T1048 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453256 T1135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f1f51d name:ZooKeeperConnection Watcher:127.0.0.1:39604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453256 T1048 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453258 T1048 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 453263 T1095 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 453265 T1070 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 453267 T1072 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 453274 T1072 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 454481 T1071 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 454482 T1071 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 454482 T1071 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 454486 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1130 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454486 T1135 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 454578 T1072 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 454578 T1072 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:50024_j_mm%2Fa
   [junit4]   2> 454579 T1072 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:39878_j_mm%2Fa
   [junit4]   2> 454579 T1072 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:60103_j_mm%2Fa
   [junit4]   2> 454580 T1072 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:52277_j_mm%2Fa
   [junit4]   2> 454582 T1098 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 454583 T1111 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 454582 T1083 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 454583 T1063 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 454584 T1098 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 454585 T1111 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 454585 T1083 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 454585 T1063 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 455994 T1071 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 455995 T1071 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50024/j_mm/a",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50024_j_mm%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 455998 T1071 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60103/j_mm/a",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60103_j_mm%2Fa",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 456001 T1071 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39878/j_mm/a",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39878_j_mm%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 456012 T1071 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52277/j_mm/a",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52277_j_mm%2Fa",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 456017 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1130 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456017 T1135 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 456584 T1098 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 456585 T1111 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 456585 T1083 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 456585 T1098 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1
   [junit4]   2> 456586 T1111 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1
   [junit4]   2> 456586 T1111 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 456586 T1083 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2
   [junit4]   2> 456587 T1083 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 456587 T1111 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 456586 T1098 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 456586 T1063 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 456588 T1083 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 456589 T1098 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 456589 T1063 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2
   [junit4]   2> 456589 T1063 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 456588 T1111 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 456589 T1098 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 456590 T1063 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 456589 T1083 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 456590 T1063 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 456591 T1111 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 456591 T1098 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 456591 T1083 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 456591 T1111 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 456592 T1083 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 456592 T1063 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 456591 T1098 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 456592 T1063 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 456594 T1083 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 456594 T1111 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 456594 T1098 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 456595 T1083 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/'
   [junit4]   2> 456595 T1111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/'
   [junit4]   2> 456595 T1098 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/'
   [junit4]   2> 456597 T1063 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 456599 T1063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/'
   [junit4]   2> 456659 T1063 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 456660 T1098 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 456673 T1083 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 456679 T1111 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 456707 T1098 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 456714 T1098 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 456720 T1098 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 456732 T1063 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 456734 T1063 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 456745 T1083 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 456748 T1083 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 456748 T1063 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 456760 T1111 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 456762 T1111 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 456762 T1083 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 456781 T1111 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 457378 T1098 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 457384 T1098 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 457411 T1098 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 457422 T1098 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457430 T1098 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457433 T1098 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457435 T1098 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457436 T1098 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457436 T1098 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457437 T1098 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457437 T1098 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457438 T1098 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 457438 T1098 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/data/
   [junit4]   2> 457438 T1098 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ded9a
   [junit4]   2> 457439 T1098 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/data
   [junit4]   2> 457439 T1098 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 457439 T1098 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 457440 T1098 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383674001166/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 457440 T1098 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=94.2587890625, floorSegmentMB=0.5771484375, forceMergeDeletesPctAllowed=25.438199744044518, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 457446 T1098 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a9543c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6e462d),segFN=segments_1,generation=1}
   [junit4]   2> 457446 T1098 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457449 T1098 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 457449 T1098 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 457450 T1098 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 457450 T1098 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457450 T1098 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457450 T1098 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 457451 T1098 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457451 T1098 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457451 T1098 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 457452 T1098 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 457452 T1098 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 457452 T1098 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 457453 T1098 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 457453 T1098 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 457454 T1098 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 457456 T1098 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 457466 T1098 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 457467 T1098 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 457467 T1098 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=58.0615234375, floorSegmentMB=1.638671875, forceMergeDeletesPctAllowed=1.159585173215839, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 457468 T1098 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a9543c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6e462d),segFN=segments_1,generation=1}
   [junit4]   2> 457468 T1098 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457468 T1098 oass.SolrIndexSearcher.<init> Opening Searcher@1ebd911 main
   [junit4]   2> 457471 T1140 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@1ebd911 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 457472 T1098 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 457472 T1098 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:50024/j_mm/a collection:delLiveColl shard:shard1
   [junit4]   2> 457474 T1098 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 457494 T1098 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 457497 T1098 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 457520 T1063 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 457527 T1071 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 457532 T1075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1102 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1130 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457532 T1135 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457534 T1083 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 457535 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 457535 T1063 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 457547 T1083 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 457549 T1063 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 457555 T1083 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 457563 T1111 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 457568 T1063 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457570 T1083 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457573 T1063 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457574 T1111 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 457575 T1083 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457577 T1063 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457584 T1111 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 457584 T1063 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457585 T1063 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457585 T1063 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457585 T1083 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457586 T1063 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457587 T1063 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457587 T1063 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 457588 T1083 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457588 T1083 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457588 T1063 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/data/
   [junit4]   2> 457588 T1083 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457589 T1063 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ded9a
   [junit4]   2> 457590 T1063 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/data
   [junit4]   2> 457590 T1063 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/data/index/
   [junit4]   2> 457591 T1063 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 457590 T1083 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457592 T1083 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457591 T1063 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383673999767/delLiveColl_shard2_replica2/data/index
   [junit4]   2> 457592 T1083 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 457592 T1063 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=94.2587890625, floorSegmentMB=0.5771484375, forceMergeDeletesPctAllowed=25.438199744044518, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 457592 T1083 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/data/
   [junit4]   2> 457593 T1063 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@82d17c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d30e49),segFN=segments_1,generation=1}
   [junit4]   2> 457594 T1083 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ded9a
   [junit4]   2> 457594 T1063 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457597 T1083 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/data
   [junit4]   2> 457598 T1083 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 457598 T1111 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457598 T1083 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 457599 T1063 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 457600 T1083 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383673999994/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 457600 T1063 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 457600 T1083 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=94.2587890625, floorSegmentMB=0.5771484375, forceMergeDeletesPctAllowed=25.438199744044518, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 457601 T1063 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 457601 T1063 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457601 T1083 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@cbc983 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5c37bc),segFN=segments_1,generation=1}
   [junit4]   2> 457602 T1083 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457602 T1063 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457602 T1111 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 457602 T1063 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 457603 T1063 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457604 T1063 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457604 T1063 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 457605 T1063 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 457606 T1063 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 457606 T1063 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 457606 T1083 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 457606 T1111 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457606 T1063 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 457607 T1083 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 457608 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457608 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457607 T1063 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 457608 T1111 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 457609 T1063 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 457608 T1083 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 457609 T1083 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457610 T1083 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 457610 T1083 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 457610 T1083 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457610 T1063 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 457611 T1083 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 457611 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 457611 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 457611 T1083 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 457612 T1111 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 457612 T1083 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 457612 T1111 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/data/
   [junit4]   2> 457613 T1083 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 457613 T1083 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 457613 T1111 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ded9a
   [junit4]   2> 457613 T1083 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 457614 T1063 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 457614 T1111 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/data
   [junit4]   2> 457614 T1063 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 457614 T1083 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 457614 T1111 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 457615 T1083 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 457615 T1063 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=58.0615234375, floorSegmentMB=1.638671875, forceMergeDeletesPctAllowed=1.159585173215839, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 457615 T1111 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 457616 T1063 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@82d17c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d30e49),segFN=segments_1,generation=1}
   [junit4]   2> 457617 T1083 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 457616 T1111 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383674002364/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 457617 T1063 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457617 T1111 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=94.2587890625, floorSegmentMB=0.5771484375, forceMergeDeletesPctAllowed=25.438199744044518, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 457618 T1063 oass.SolrIndexSearcher.<init> Opening Searcher@3a3f8b main
   [junit4]   2> 457618 T1111 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@333eda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1a437e),segFN=segments_1,generation=1}
   [junit4]   2> 457619 T1111 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 457620 T1083 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 457620 T1083 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 457621 T1083 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=58.0615234375, floorSegmentMB=1.638671875, forceMergeDeletesPctAllowed=1.159585173215839, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 457621 T1141 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica2] Registered new searcher Searcher@3a3f8b main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 457621 T1083 oasc.SolrDeletionP

[...truncated too long message...]

[junit4]   2> 
   [junit4]   2> 575756 T1134 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 575757 T1134 oaz.ClientCnxn$SendThread.run WARN Session 0x1422965b0b7000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 576256 T1134 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 576258 T1134 oaz.ClientCnxn$SendThread.run WARN Session 0x1422965b0b7000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 576756 T1134 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 577257 T1134 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 577257 T1134 oaz.ClientCnxn$SendThread.run WARN Session 0x1422965b0b7000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 129701 T1047 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=sq_AL, timezone=Asia/Muscat
   [junit4]   2> NOTE: Linux 3.8.0-32-generic i386/Oracle Corporation 1.7.0_45 (32-bit)/cpus=8,threads=3,free=17742200,total=129241088
   [junit4]   2> NOTE: All tests run in this JVM: [DirectUpdateHandlerTest, TestInfoStreamLogging, UniqFieldsUpdateProcessorFactoryTest, TestStressUserVersions, TestBinaryResponseWriter, TestPhraseSuggestions, TestCloudManagedSchemaAddField, MultiTermTest, TestAnalyzedSuggestions, SolrPluginUtilsTest, TimeZoneUtilsTest, TestManagedSchemaFieldResource, ShardRoutingCustomTest, TestMultiCoreConfBootstrap, IndexSchemaRuntimeFieldTest, TestShardHandlerFactory, TestRandomFaceting, OpenCloseCoreStressTest, FileUtilsTest, ConvertedLegacyTest, ZkNodePropsTest, CoreMergeIndexesAdminHandlerTest, DirectSolrSpellCheckerTest, SpellCheckComponentTest, TestArbitraryIndexDir, CachingDirectoryFactoryTest, SolrRequestParserTest, TestFastOutputStream, TestCSVLoader, TestFastWriter, HighlighterTest, CoreAdminCreateDiscoverTest, TestRecovery, XsltUpdateRequestHandlerTest, TestStressRecovery, TestPartialUpdateDeduplication, BadIndexSchemaTest, TestLazyCores, TestSolrDeletionPolicy2, NoCacheHeaderTest, TestSerializedLuceneMatchVersion, DistanceFunctionTest, TestFieldTypeCollectionResource, SpellingQueryConverterTest, TestFuzzyAnalyzedSuggestions, TestSolrQueryParserDefaultOperatorResource, CurrencyFieldOpenExchangeTest, HighlighterConfigTest, TestFiltering, TestSolrXmlPersistor, DebugComponentTest, TestFoldingMultitermQuery, DocumentAnalysisRequestHandlerTest, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, IndexBasedSpellCheckerTest, TestComponentsName, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=1C813DED28B0406D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sq_AL -Dtests.timezone=Asia/Muscat -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    >    2) Thread[id=1135, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1134, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[1C813DED28B0406D]-SendThread(localhost.localdomain:39604), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1C813DED28B0406D]:0)
   [junit4] Completed on J1 in 129.75s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 620 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 333 suites, 1463 tests, 3 suite-level errors, 1 error, 818 ignored (4 assumptions)

Total time: 34 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8193 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8193/
Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node3 in shard/collection : shard1/delLiveColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node3 in shard/collection : shard1/delLiveColl
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966:720CA955DC83F95A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:149)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:135)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)




Build Log:
[...truncated 10744 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 1316998 T3221 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1317001 T3221 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-DeleteReplicaTest-1383706308253
   [junit4]   2> 1317002 T3221 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1317003 T3222 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1317103 T3221 oasc.ZkTestServer.run start zk server on port:53963
   [junit4]   2> 1317114 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317198 T3228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bf416ae name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317198 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317198 T3221 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1317203 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317204 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67cfe1c5 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317205 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317205 T3221 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1317208 T3221 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1317210 T3221 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1317218 T3221 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1317224 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1317225 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1317228 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1317229 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1317231 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1317232 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1317234 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1317234 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1317237 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1317237 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1317239 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1317240 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1317242 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1317242 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1317245 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1317245 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1317247 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1317247 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1317250 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1317250 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1317252 T3221 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1317252 T3221 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1317315 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1317319 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46383
   [junit4]   2> 1317320 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506
   [junit4]   2> 1317320 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/'
   [junit4]   2> 1317329 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/solr.xml
   [junit4]   2> 1317358 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1317358 T3221 oasc.CoreContainer.<init> New CoreContainer 1178080130
   [junit4]   2> 1317358 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/]
   [junit4]   2> 1317359 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1317360 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1317361 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1317362 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1317362 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1317362 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1317363 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1317363 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317366 T3241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a0b2d2 name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317366 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317368 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317370 T3243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b4c53af name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317370 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317372 T3221 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1317375 T3221 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1317377 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1317378 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46383_
   [junit4]   2> 1317379 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46383_
   [junit4]   2> 1317381 T3221 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1317385 T3221 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1317387 T3221 oasc.Overseer.start Overseer (id=90682576617865219-127.0.0.1:46383_-n_0000000000) starting
   [junit4]   2> 1317389 T3221 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1317392 T3245 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1317392 T3221 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1317402 T3221 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1317405 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1317407 T3244 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1317409 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506
   [junit4]   2> 1317409 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/conf/
   [junit4]   2> 1317438 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1317438 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1317439 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1317440 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317442 T3248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6516790d name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317442 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317456 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1317458 T3221 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1317517 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1317518 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52279
   [junit4]   2> 1317519 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709
   [junit4]   2> 1317519 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/'
   [junit4]   2> 1317529 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/solr.xml
   [junit4]   2> 1317561 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1317562 T3221 oasc.CoreContainer.<init> New CoreContainer 1561880160
   [junit4]   2> 1317562 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/]
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1317563 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1317564 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1317565 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1317565 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1317566 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1317566 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1317566 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1317567 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1317567 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317569 T3259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@436f2ac1 name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317569 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317572 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1317573 T3261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42d67075 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1317573 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1317578 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1318581 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52279_
   [junit4]   2> 1318582 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52279_
   [junit4]   2> 1318585 T3243 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318585 T3261 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318585 T3248 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1318590 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709
   [junit4]   2> 1318592 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/conf/
   [junit4]   2> 1318619 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1318620 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1318620 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1318679 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1318681 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53439
   [junit4]   2> 1318681 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872
   [junit4]   2> 1318682 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/'
   [junit4]   2> 1318691 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/solr.xml
   [junit4]   2> 1318730 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1318731 T3221 oasc.CoreContainer.<init> New CoreContainer 2101110435
   [junit4]   2> 1318731 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/]
   [junit4]   2> 1318732 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1318732 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1318733 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1318734 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1318735 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1318736 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1318736 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1318737 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1318737 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1318738 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1318739 T3273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a4e869e name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1318740 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1318742 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1318743 T3275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d1c058 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1318744 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1318747 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1319749 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53439_
   [junit4]   2> 1319751 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53439_
   [junit4]   2> 1319754 T3261 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319754 T3248 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319754 T3275 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319755 T3243 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1319758 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872
   [junit4]   2> 1319760 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383706309872/conf/
   [junit4]   2> 1319786 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1319787 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1319787 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1319844 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1319846 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35150
   [junit4]   2> 1319847 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039
   [junit4]   2> 1319847 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/'
   [junit4]   2> 1319856 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/solr.xml
   [junit4]   2> 1319883 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1319884 T3221 oasc.CoreContainer.<init> New CoreContainer 1596363683
   [junit4]   2> 1319884 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/]
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1319885 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1319886 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1319887 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1319888 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1319888 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1319888 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1319889 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1319890 T3287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44f218a name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1319890 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1319893 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1319895 T3289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f3776cd name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1319895 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1319899 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1320901 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35150_
   [junit4]   2> 1320902 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35150_
   [junit4]   2> 1320905 T3261 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3248 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320905 T3275 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3289 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320906 T3243 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1320910 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039
   [junit4]   2> 1320911 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/conf/
   [junit4]   2> 1320931 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1320931 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1320932 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1320995 T3221 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1320998 T3221 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36332
   [junit4]   2> 1320999 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1320999 T3221 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1321000 T3221 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184
   [junit4]   2> 1321000 T3221 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/'
   [junit4]   2> 1321010 T3221 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/solr.xml
   [junit4]   2> 1321038 T3221 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1321038 T3221 oasc.CoreContainer.<init> New CoreContainer 695590024
   [junit4]   2> 1321039 T3221 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/]
   [junit4]   2> 1321039 T3221 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1321040 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1321041 T3221 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1321042 T3221 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1321042 T3221 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1321043 T3221 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53963/solr
   [junit4]   2> 1321043 T3221 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1321044 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1321045 T3301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199f332c name:ZooKeeperConnection Watcher:127.0.0.1:53963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1321045 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1321048 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1321049 T3303 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ef8bd2 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1321049 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1321052 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322055 T3221 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36332_
   [junit4]   2> 1322056 T3221 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36332_
   [junit4]   2> 1322059 T3248 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3261 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322060 T3243 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3303 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322059 T3275 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322060 T3289 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1322064 T3221 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184
   [junit4]   2> 1322065 T3221 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/conf/
   [junit4]   2> 1322093 T3221 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1322093 T3221 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1322094 T3221 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1322100 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1322102 T3306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9f73ad name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1322103 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1322104 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322113 T3221 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1322114 T3308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@663cec84 name:ZooKeeperConnection Watcher:127.0.0.1:53963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1322114 T3221 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1322115 T3221 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1322118 T3236 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 1322120 T3243 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1322121 T3245 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1322121 T3245 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1323418 T3244 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1323419 T3244 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1323419 T3244 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1323423 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323423 T3308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1323526 T3245 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1323526 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:35150_
   [junit4]   2> 1323527 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:46383_
   [junit4]   2> 1323527 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:36332_
   [junit4]   2> 1323528 T3245 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:52279_
   [junit4]   2> 1323530 T3284 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 1323530 T3255 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 1323530 T3238 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 1323531 T3298 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 1323532 T3284 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3238 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3298 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1323532 T3255 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1324939 T3244 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1324939 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35150",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35150_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324944 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52279",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52279_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324946 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46383",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46383_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324949 T3244 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36332",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36332_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1324996 T3308 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3303 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1324996 T3306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1325532 T3284 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1325532 T3238 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1325533 T3255 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1325533 T3284 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1
   [junit4]   2> 1325533 T3298 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1325534 T3255 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2
   [junit4]   2> 1325533 T3284 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325533 T3238 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2
   [junit4]   2> 1325534 T3238 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325534 T3298 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1
   [junit4]   2> 1325534 T3255 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325535 T3238 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325535 T3284 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325535 T3298 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1325536 T3284 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325536 T3255 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325537 T3255 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325537 T3284 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325536 T3238 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325537 T3284 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325538 T3255 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325538 T3255 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325536 T3298 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 1325539 T3298 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 1325539 T3238 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 1325540 T3238 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325541 T3284 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325541 T3298 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1325542 T3255 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325542 T3238 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325543 T3284 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/'
   [junit4]   2> 1325544 T3298 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1325544 T3238 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/'
   [junit4]   2> 1325545 T3298 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/'
   [junit4]   2> 1325546 T3255 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/'
   [junit4]   2> 1325599 T3284 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325600 T3298 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325604 T3238 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325621 T3255 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1325644 T3284 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325645 T3298 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325645 T3284 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325646 T3298 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325652 T3298 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1325652 T3284 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1325664 T3238 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325666 T3238 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325677 T3238 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1325693 T3255 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1325694 T3255 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1325706 T3255 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1326169 T3238 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326175 T3238 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326175 T3298 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326177 T3238 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326180 T3298 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326183 T3298 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326190 T3238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326194 T3238 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326197 T3284 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326198 T3238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326199 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326202 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326203 T3238 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326204 T3284 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326207 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326208 T3238 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326208 T3238 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326208 T3284 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326208 T3238 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/
   [junit4]   2> 1326209 T3238 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326210 T3298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326210 T3238 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data
   [junit4]   2> 1326211 T3238 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 1326211 T3238 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326212 T3238 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383706308506/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 1326213 T3238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326214 T3238 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c31b185 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40dc859a),segFN=segments_1,generation=1}
   [junit4]   2> 1326214 T3238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326215 T3298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326221 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326222 T3238 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326223 T3238 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326223 T3238 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326223 T3298 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326224 T3238 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326224 T3238 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326225 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326226 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326225 T3238 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326226 T3298 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326228 T3238 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326229 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326229 T3298 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326230 T3298 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326230 T3298 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/
   [junit4]   2> 1326230 T3284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326230 T3298 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326232 T3298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data
   [junit4]   2> 1326232 T3298 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 1326232 T3298 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326233 T3238 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326233 T3298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383706312184/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 1326233 T3298 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326235 T3298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7a00106d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f31458d),segFN=segments_1,generation=1}
   [junit4]   2> 1326235 T3298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326236 T3284 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326236 T3238 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326237 T3238 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326237 T3238 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326238 T3238 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c31b185 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40dc859a),segFN=segments_1,generation=1}
   [junit4]   2> 1326238 T3238 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326239 T3238 oass.SolrIndexSearcher.<init> Opening Searcher@382a68f0 main
   [junit4]   2> 1326239 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326240 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326240 T3284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326241 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326241 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326242 T3298 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326242 T3313 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@382a68f0 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326242 T3298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326243 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326243 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326243 T3298 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326244 T3284 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326243 T3238 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 1326244 T3298 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326244 T3238 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:46383 collection:delLiveColl shard:shard1
   [junit4]   2> 1326245 T3298 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326245 T3298 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326245 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326246 T3238 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 1326246 T3284 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326246 T3298 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326246 T3284 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326247 T3298 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326247 T3284 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/
   [junit4]   2> 1326247 T3298 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326248 T3284 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326249 T3284 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data
   [junit4]   2> 1326249 T3298 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326249 T3284 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 1326250 T3284 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326256 T3284 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383706311039/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 1326257 T3284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326257 T3284 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a276ebf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8214ea8),segFN=segments_1,generation=1}
   [junit4]   2> 1326258 T3284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326258 T3298 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326258 T3298 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326259 T3298 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326259 T3298 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7a00106d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f31458d),segFN=segments_1,generation=1}
   [junit4]   2> 1326260 T3298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326260 T3298 oass.SolrIndexSearcher.<init> Opening Searcher@7a78aa9a main
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326261 T3238 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1326261 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1326262 T3314 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@7a78aa9a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326262 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1326263 T3284 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326263 T3298 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 1326263 T3284 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1326263 T3298 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:36332 collection:delLiveColl shard:shard2
   [junit4]   2> 1326264 T3238 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 1326264 T3284 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1326264 T3284 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1326265 T3298 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1326265 T3284 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1326266 T3284 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1326266 T3284 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1326267 T3284 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1326270 T3284 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1326270 T3284 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1326270 T3298 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1326270 T3284 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1326271 T3284 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a276ebf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8214ea8),segFN=segments_1,generation=1}
   [junit4]   2> 1326271 T3284 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326272 T3284 oass.SolrIndexSearcher.<init> Opening Searcher@6f609d41 main
   [junit4]   2> 1326272 T3298 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=180000
   [junit4]   2> 1326274 T3315 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@6f609d41 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1326275 T3284 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 1326275 T3284 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:35150 collection:delLiveColl shard:shard1
   [junit4]   2> 1326288 T3255 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1326295 T3255 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1326298 T3255 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1326311 T3255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326315 T3255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1326319 T3255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326321 T3255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1326323 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1326323 T3255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1326323 T3255 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1326324 T3255 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/
   [junit4]   2> 1326324 T3255 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c8e1a67
   [junit4]   2> 1326325 T3255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data
   [junit4]   2> 1326325 T3255 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index/
   [junit4]   2> 1326326 T3255 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1326326 T3255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383706308709/delLiveColl_shard2_replica2/data/index
   [junit4]   2> 1326327 T3255 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=8.3662109375, floorSegmentMB=1.677734375, forceMergeDeletesPctAllowed=21.828590384409768, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1326328 T3255 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3a2f7124 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3eee2ac5),segFN=segments_1,generation=1}
   [junit4]   2> 1326328 T3255 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1326332 T3255 oasup.UpdateRequestProcessorChain.init creating updateRequestProces

[...truncated too long message...]

 1419881 T3307 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 1420381 T3307 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 1420382 T3307 oaz.ClientCnxn$SendThread.run WARN Session 0x1422b52aea0000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 103884 T3220 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=zh_HK, timezone=America/Indiana/Tell_City
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=3,free=378498024,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestQueryTypes, TestSolr4Spatial, ShardSplitTest, PathHierarchyTokenizerFactoryTest, DirectSolrConnectionTest, LeaderElectionIntegrationTest, SpellCheckComponentTest, TestQuerySenderNoQuery, SimplePostToolTest, ZkNodePropsTest, TestIndexingPerformance, TestQuerySenderListener, SuggesterWFSTTest, TestCoreDiscovery, TestStressReorder, StandardRequestHandlerTest, BJQParserTest, TestArbitraryIndexDir, TestCSVResponseWriter, AddSchemaFieldsUpdateProcessorFactoryTest, TestWordDelimiterFilterFactory, TestComponentsName, TestSolrJ, TestIBSimilarityFactory, RequiredFieldsTest, TestShardHandlerFactory, TestDynamicFieldCollectionResource, TestRangeQuery, TestFastOutputStream, TestCloudManagedSchemaAddField, HdfsChaosMonkeySafeLeaderTest, SortByFunctionTest, TestDFRSimilarityFactory, TestLazyCores, BadIndexSchemaTest, TestRecoveryHdfs, SolrRequestParserTest, AlternateDirectoryTest, TestSchemaResource, PluginInfoTest, RAMDirectoryFactoryTest, ChaosMonkeySafeLeaderTest, DateFieldTest, ZkSolrClientTest, TestSweetSpotSimilarityFactory, DistanceFunctionTest, TestNumberUtils, TestIndexSearcher, PrimUtilsTest, TestAtomicUpdateErrorCases, TestDistribDocBasedVersion, BlockCacheTest, TestAnalyzedSuggestions, FileUtilsTest, DocumentAnalysisRequestHandlerTest, TestStressRecovery, OpenExchangeRatesOrgProviderTest, DeleteInactiveReplicaTest, JSONWriterTest, SolrCoreCheckLockOnStartupTest, ConvertedLegacyTest, CurrencyFieldOpenExchangeTest, TestCollapseQParserPlugin, TestCSVLoader, StressHdfsTest, StatsComponentTest, TestFiltering, CollectionsAPIDistributedZkTest, TestUpdate, SolrCmdDistributorTest, DirectSolrSpellCheckerTest, IndexSchemaRuntimeFieldTest, HdfsBasicDistributedZk2Test, CoreMergeIndexesAdminHandlerTest, SpellCheckCollatorTest, TestFuzzyAnalyzedSuggestions, FullSolrCloudDistribCmdsTest, TestDefaultSimilarityFactory, CachingDirectoryFactoryTest, UniqFieldsUpdateProcessorFactoryTest, BasicZkTest, TestZkChroot, TestRandomDVFaceting, TestDistributedSearch, TestFaceting, TestHashPartitioner, ZkControllerTest, TestJoin, PeerSyncTest, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, TestFoldingMultitermQuery, SuggesterTSTTest, NoCacheHeaderTest, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestWriterPerf, TestReversedWildcardFilterFactory, TestOmitPositions, RequestHandlersTest, TestValueSourceCache, MoreLikeThisHandlerTest, FastVectorHighlighterTest, TestJmxIntegration, ReturnFieldsTest, MBeansHandlerTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, SearchHandlerTest, HighlighterConfigTest, ResponseLogComponentTest, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, MinimalSchemaTest, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCharFilters, SynonymTokenizerTest, EchoParamsTest, TestLMDirichletSimilarityFactory, TimeZoneUtilsTest, TestSystemIdResolver, SpellingQueryConverterTest, DOMUtilTest, TestDocumentBuilder, TestRTGBase, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=F3EA274DABDC9966 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK -Dtests.timezone=America/Indiana/Tell_City -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=3308, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3307, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[F3EA274DABDC9966]-SendThread(localhost.localdomain:53963), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F3EA274DABDC9966]:0)
   [junit4] Completed on J1 in 103.94s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 3 suite-level errors, 1 error, 35 ignored (6 assumptions)

Total time: 44 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8192 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8192/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate

Error Message:
expected:<[in]active> but was:<[]active>

Stack Trace:
org.junit.ComparisonFailure: expected:<[in]active> but was:<[]active>
	at __randomizedtesting.SeedInfo.seed([4289FDB0D725E1A4:92EB70FC89F03B20]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate(SliceStateUpdateTest.java:183)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10062 lines...]
   [junit4] Suite: org.apache.solr.cloud.SliceStateUpdateTest
   [junit4]   2> 415330 T1199 oas.SolrTestCaseJ4.setUp ###Starting testSliceStateUpdate
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233
   [junit4]   2> 415330 T1199 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 415331 T1200 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 415431 T1199 oasc.ZkTestServer.run start zk server on port:51293
   [junit4]   2> 415432 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415520 T1206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4347a370 name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415520 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415520 T1199 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 415525 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415526 T1208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c438065 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415526 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415526 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 415531 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 415534 T1199 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 415536 T1199 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 415538 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 415538 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 415541 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 415542 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 415544 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 415544 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 415546 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 415546 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 415548 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 415548 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 415550 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 415550 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 415552 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 415553 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 415555 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 415555 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 415557 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 415557 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 415559 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 415559 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 415561 T1199 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 415562 T1199 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 415564 T1199 oasc.SliceStateUpdateTest.setUp ####SETUP_START testSliceStateUpdate
   [junit4]   2> 415565 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415566 T1210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70a2114 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415566 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415566 T1199 oascc.SolrZkClient.makePath makePath: /collections/testcore
   [junit4]   2> 415569 T1199 oascc.SolrZkClient.makePath makePath: /collections/testcore/shards
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 415574 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 415584 T1199 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 415618 T1199 oasc.CoreContainer.<init> New CoreContainer 915571128
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr
   [junit4]   2> 415618 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/'
   [junit4]   2> 415628 T1199 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/solr.xml
   [junit4]   2> 415659 T1199 oasc.CoreContainer.<init> New CoreContainer 22158319
   [junit4]   2> 415659 T1199 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 415660 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 415661 T1199 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 415662 T1199 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 415662 T1199 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 415663 T1199 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51293/solr
   [junit4]   2> 415663 T1199 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 415664 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415665 T1212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6236bbb6 name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415665 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415668 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 415669 T1214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1313a56f name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 415669 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 415671 T1199 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 415674 T1199 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 415676 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 415678 T1199 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 415679 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 415681 T1199 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 415684 T1199 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 415686 T1199 oasc.Overseer.start Overseer (id=90682159414312964-127.0.0.1:8983_solr-n_0000000000) starting
   [junit4]   2> 415691 T1199 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 415694 T1216 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 415695 T1199 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 415697 T1199 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 415698 T1199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 415700 T1215 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 415702 T1217 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 415702 T1217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 415703 T1217 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 417205 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 417206 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 417206 T1215 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 417212 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 417703 T1217 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 417704 T1217 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 417704 T1217 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 417705 T1217 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 417706 T1217 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 417708 T1217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 417709 T1217 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 417709 T1217 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 417767 T1217 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 417811 T1217 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 417813 T1217 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 417819 T1217 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 418122 T1217 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 418127 T1217 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 418129 T1217 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 418148 T1217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 418152 T1217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 418156 T1217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 418158 T1217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 418160 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 418160 T1217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 418160 T1217 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 418161 T1217 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/
   [junit4]   2> 418161 T1217 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30eb5ea0
   [junit4]   2> 418162 T1217 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 418162 T1217 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index/
   [junit4]   2> 418163 T1217 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index' doesn't exist. Creating new index...
   [junit4]   2> 418163 T1217 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 418164 T1217 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1997227149, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 418165 T1217 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7c4f7bc8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20704adf)),segFN=segments_1,generation=1}
   [junit4]   2> 418166 T1217 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 418166 T1217 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 418171 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 418172 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 418172 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 418173 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 418173 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 418174 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 418174 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 418175 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 418175 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 418176 T1217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 418177 T1217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 418177 T1217 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 418177 T1217 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 418178 T1217 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 418179 T1217 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 418180 T1217 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 418181 T1217 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 418183 T1217 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 418183 T1217 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 418184 T1217 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 418185 T1217 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 418185 T1217 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 418186 T1217 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 418187 T1217 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418188 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418189 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418190 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418191 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418192 T1217 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418192 T1217 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418193 T1217 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 418194 T1217 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 418194 T1217 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 418196 T1217 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 418197 T1217 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 418198 T1217 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [ENUMSCONFIG.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML]
   [junit4]   2> 418200 T1217 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 418200 T1217 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 418201 T1217 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 418201 T1217 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7c4f7bc8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20704adf)),segFN=segments_1,generation=1}
   [junit4]   2> 418201 T1217 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 418201 T1217 oass.SolrIndexSearcher.<init> Opening Searcher@4826a22e main
   [junit4]   2> 418202 T1217 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 418205 T1217 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 418224 T1218 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4826a22e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 418227 T1217 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 418228 T1217 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:8983/solr collection:collection1 shard:shard1
   [junit4]   2> 418229 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 418234 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 418236 T1217 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:8983/solr/collection1/ shard1
   [junit4]   2> 418236 T1217 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 418719 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 418729 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 418741 T1217 oasc.ZkController.register We are http://127.0.0.1:8983/solr/collection1/ and leader is http://127.0.0.1:8983/solr/collection1/
   [junit4]   2> 418742 T1217 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:8983/solr
   [junit4]   2> 418742 T1217 oasc.ZkController.checkRecovery WARN Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 418742 T1217 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 418742 T1217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 418744 T1217 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 418745 T1199 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/]
   [junit4]   2> 418746 T1199 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 418746 T1199 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 418747 T1199 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 418748 T1199 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 418748 T1199 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 418748 T1199 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 418748 T1199 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 418749 T1199 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51293/solr
   [junit4]   2> 418749 T1199 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 418749 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418751 T1221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31ca775e name:ZooKeeperConnection Watcher:127.0.0.1:51293 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418751 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418754 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 418761 T1223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2398635e name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 418761 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 418765 T1199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 419768 T1199 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 419770 T1199 oasc.ZkController.createEphemeralLiveNode Found a previous node that still exists while trying to register a new live node /live_nodes/127.0.0.1:8983_solr - removing existing node to create another.
   [junit4]   2> 419770 T1199 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:8983_solr
   [junit4]   2> 419770 T1214 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 419770 T1223 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 419772 T1214 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 419773 T1223 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 419778 T1224 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 419778 T1224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 419779 T1224 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 419779 T1224 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 419779 T1224 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1
   [junit4]   2> 419779 T1224 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 419780 T1224 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 419780 T1224 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/'
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 419781 T1224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 419819 T1224 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 419862 T1224 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 419864 T1224 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 419870 T1224 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 420170 T1224 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 420175 T1224 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 420177 T1224 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 420189 T1224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 420192 T1224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 420195 T1224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 420196 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 420196 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 420197 T1224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 420198 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 420198 T1224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 420198 T1224 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 420198 T1224 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/
   [junit4]   2> 420198 T1224 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30eb5ea0
   [junit4]   2> 420199 T1224 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 420199 T1224 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index/
   [junit4]   2> 420199 T1224 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index' doesn't exist. Creating new index...
   [junit4]   2> 420199 T1224 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 420200 T1224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1997227149, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 420200 T1224 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@33d633b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75471931)),segFN=segments_1,generation=1}
   [junit4]   2> 420201 T1224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 420201 T1224 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 420204 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 420205 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 420206 T1224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 420206 T1224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 420206 T1224 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 420206 T1224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 420206 T1224 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 420207 T1224 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 420207 T1224 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 420208 T1224 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 420209 T1224 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 420210 T1224 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420210 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420211 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420212 T1224 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420212 T1224 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 420213 T1224 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 420215 T1224 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 420217 T1224 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 420218 T1224 oasc.SolrCore.initDeprecatedSupport WARN adding ShowFileRequestHandler with hidden files: [ENUMSCONFIG.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT, SOLRCONFIG.SNIPPET.RANDOMINDEXCONFIG.XML]
   [junit4]   2> 420219 T1224 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 420220 T1224 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 420220 T1224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=40, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 420220 T1224 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@33d633b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@75471931)),segFN=segments_1,generation=1}
   [junit4]   2> 420221 T1224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 420221 T1224 oass.SolrIndexSearcher.<init> Opening Searcher@b589bb5 main
   [junit4]   2> 420221 T1224 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 420222 T1224 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 420230 T1225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b589bb5 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 420236 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 420236 T1224 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 420236 T1224 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:8983/solr collection:collection1 shard:shard1
   [junit4]   2> 420237 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 420240 T1224 oasc.ZkController.register We are http://127.0.0.1:8983/solr/collection1/ and leader is http://127.0.0.1:8983/solr/collection1/
   [junit4]   2> 420240 T1224 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:8983/solr
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 420240 T1224 oasc.ZkController.checkRecovery WARN Skipping recovery according to sys prop solrcloud.skip.autorecovery
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 420240 T1224 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 420240 T1224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 420240 T1215 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 420242 T1224 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 420242 T1199 oasc.SliceStateUpdateTest.setUp ####SETUP_END testSliceStateUpdate
   [junit4]   2> 420243 T1199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 420244 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 420245 T1228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b72bd6 name:ZooKeeperConnection Watcher:127.0.0.1:51293/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 420245 T1199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 420246 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 420246 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 420249 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 470260 T1199 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=915571128
   [junit4]   2> 471335 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 471336 T1215 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 471336 T1215 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 471344 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 471344 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 473261 T1199 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 473262 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 473263 T1199 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6ffd9f54
   [junit4]   2> 473277 T1199 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 473277 T1199 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 473277 T1199 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 473277 T1199 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 473279 T1199 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2;done=false>>]
   [junit4]   2> 473279 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 473280 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index;done=false>>]
   [junit4]   2> 473280 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 473281 T1215 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90682159414312964-127.0.0.1:8983_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 473283 T1199 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22158319
   [junit4]   2> 473284 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 473285 T1223 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 473286 T1223 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:8983/solr/collection1/ shard1
   [junit4]   2> 473286 T1223 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 473295 T1223 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 473297 T1223 oasc.Overseer.start Overseer (id=90682159414312966-127.0.0.1:8983_solr-n_0000000001) starting
   [junit4]   2> 473300 T1230 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 473301 T1229 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 473302 T1229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 473303 T1229 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:8983/solr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:8983_solr",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":null,
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 473311 T1223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 474284 T1199 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 474284 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 474286 T1199 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6ace3348
   [junit4]   2> 474300 T1199 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 474300 T1199 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 474300 T1199 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 474301 T1199 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 474302 T1199 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2;done=false>>]
   [junit4]   2> 474303 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2
   [junit4]   2> 474304 T1199 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index;done=false>>]
   [junit4]   2> 474304 T1199 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SliceStateUpdateTest-1383699942233/data2/index
   [junit4]   2> 474306 T1229 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90682159414312966-127.0.0.1:8983_solr-n_0000000001) am no longer a leader.
   [junit4]   2> 474310 T1199 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51293 51293
   [junit4]   2> 474606 T1199 oas.SolrTestCaseJ4.tearDown ###Ending testSliceStateUpdate
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SliceStateUpdateTest -Dtests.method=testSliceStateUpdate -Dtests.seed=4289FDB0D725E1A4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_LU -Dtests.timezone=Asia/Omsk -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 59.3s J0 | SliceStateUpdateTest.testSliceStateUpdate <<<
   [junit4]    > Throwable #1: org.junit.ComparisonFailure: expected:<[in]active> but was:<[]active>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4289FDB0D725E1A4:92EB70FC89F03B20]:0)
   [junit4]    > 	at org.apache.solr.cloud.SliceStateUpdateTest.testSliceStateUpdate(SliceStateUpdateTest.java:183)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 474638 T1199 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 59317 T1198 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 475808 T1223 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=de_LU, timezone=Asia/Omsk
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=1,free=100096728,total=187760640
   [junit4]   2> NOTE: All tests run in this JVM: [TestMergePolicyConfig, TestCoreContainer, TestSolrXMLSerializer, TestSolrXmlPersistence, ResourceLoaderTest, TestConfig, TestSearchPerf, ParsingFieldUpdateProcessorsTest, QueryEqualityTest, PrimitiveFieldTypeTest, WordBreakSolrSpellCheckerTest, TestManagedSchema, TestLRUCache, BasicDistributedZkTest, TestLMDirichletSimilarityFactory, JsonLoaderTest, DirectUpdateHandlerOptimizeTest, UpdateRequestProcessorFactoryTest, DateMathParserTest, TestRealTimeGet, SyncSliceTest, UnloadDistributedZkTest, TestGroupingSearch, TestDocumentBuilder, TestFieldTypeCollectionResource, TestCharFilters, TestDocSet, LegacyHTMLStripCharFilterTest, TermsComponentTest, SynonymTokenizerTest, DocumentBuilderTest, TestDocBasedVersionConstraints, TermVectorComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestBM25SimilarityFactory, ZkControllerTest, DOMUtilTest, TestSystemIdResolver, AssignTest, TestReversedWildcardFilterFactory, SpatialFilterTest, TestFunctionQuery, TestSchemaSimilarityResource, SliceStateUpdateTest]
   [junit4] Completed on J0 in 60.59s, 1 test, 1 failure <<< FAILURES!

[...truncated 743 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 1 failure, 35 ignored (6 assumptions)

Total time: 46 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8191 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8191/
Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest:     1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)    2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)


FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Invalid replica : core_node1 in shard/collection : shard1/delLiveColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid replica : core_node1 in shard/collection : shard1/delLiveColl
	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2:E36FA8B904214FDE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:464)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:149)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:135)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:115)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 9837 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> 21331 T64 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 21340 T64 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-DeleteReplicaTest-1383694728716
   [junit4]   2> 21343 T64 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 21346 T65 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 21445 T64 oasc.ZkTestServer.run start zk server on port:38544
   [junit4]   2> 21677 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21729 T66 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 23457 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68aa419e name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23458 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23460 T64 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 23513 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23514 T66 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1422aa1fe8a0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 23528 T73 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4926aaf7 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23528 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23535 T64 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 23544 T64 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 23550 T64 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 23577 T64 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 23607 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 23621 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 23635 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 23641 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 23665 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 23665 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 23673 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 23674 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 23696 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 23697 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 23701 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 23702 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 23723 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 23724 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 23736 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 23736 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 23751 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 23752 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 23766 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 23767 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 23774 T64 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 23775 T64 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 24120 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 24346 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34437
   [junit4]   2> 24367 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 24368 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24369 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174
   [junit4]   2> 24369 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/'
   [junit4]   2> 24388 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/solr.xml
   [junit4]   2> 24455 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 24458 T64 oasc.CoreContainer.<init> New CoreContainer 968317684
   [junit4]   2> 24459 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/]
   [junit4]   2> 24475 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 24476 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 24477 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 24477 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24478 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24478 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24479 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24479 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24480 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24482 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24483 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24483 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 24495 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 24496 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24520 T85 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a97e32c name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24521 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24549 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24554 T87 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44ed040f name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24555 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24564 T64 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 24589 T64 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 24623 T64 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 24652 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34437_
   [junit4]   2> 24673 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34437_
   [junit4]   2> 24682 T64 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 24714 T64 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 24718 T64 oasc.Overseer.start Overseer (id=90681817749389315-127.0.0.1:34437_-n_0000000000) starting
   [junit4]   2> 24755 T64 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 24790 T89 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 24791 T64 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 24796 T64 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 24803 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24828 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174
   [junit4]   2> 24832 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/conf/
   [junit4]   2> 24839 T88 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 24873 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 24874 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 24874 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 24905 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 24911 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f5e8edc name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 24911 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 24923 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24934 T64 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 25067 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 25070 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36637
   [junit4]   2> 25071 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 25072 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 25072 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310
   [junit4]   2> 25073 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/'
   [junit4]   2> 25112 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/solr.xml
   [junit4]   2> 25189 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 25190 T64 oasc.CoreContainer.<init> New CoreContainer 1489186216
   [junit4]   2> 25191 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/]
   [junit4]   2> 25193 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 25193 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 25194 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 25194 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 25195 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 25195 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 25196 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 25196 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 25197 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25199 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25200 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25200 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 25201 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 25204 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25206 T103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4648548e name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25207 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25238 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 25266 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437d3a6a name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 25267 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 25274 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26285 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36637_
   [junit4]   2> 26287 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36637_
   [junit4]   2> 26293 T92 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26294 T87 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26294 T105 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 26301 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310
   [junit4]   2> 26302 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/conf/
   [junit4]   2> 26322 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 26323 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 26323 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26397 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26399 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60628
   [junit4]   2> 26400 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26400 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26401 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699
   [junit4]   2> 26401 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/'
   [junit4]   2> 26411 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/solr.xml
   [junit4]   2> 26449 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 26449 T64 oasc.CoreContainer.<init> New CoreContainer 35487695
   [junit4]   2> 26450 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/]
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 26451 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26452 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26453 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26455 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26455 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26456 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 26456 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 26457 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26459 T117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b72935a name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26460 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26465 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26468 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@521effe1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26468 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26475 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 27480 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60628_
   [junit4]   2> 27482 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60628_
   [junit4]   2> 27486 T105 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27486 T87 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27486 T119 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27487 T92 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 27492 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699
   [junit4]   2> 27493 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty2-1383694733699/conf/
   [junit4]   2> 27519 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 27520 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 27520 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 27589 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 27592 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39004
   [junit4]   2> 27592 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896
   [junit4]   2> 27593 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/'
   [junit4]   2> 27603 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/solr.xml
   [junit4]   2> 27641 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 27642 T64 oasc.CoreContainer.<init> New CoreContainer 899097207
   [junit4]   2> 27642 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/]
   [junit4]   2> 27643 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 27643 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 27644 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 27645 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 27646 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 27647 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 27647 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 27648 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 27648 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 27649 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27652 T131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63d36b29 name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27653 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27657 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 27659 T133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a102e69 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 27659 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 27665 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 28669 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39004_
   [junit4]   2> 28671 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39004_
   [junit4]   2> 28674 T87 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T119 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T92 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T133 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28675 T105 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 28681 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896
   [junit4]   2> 28682 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/conf/
   [junit4]   2> 28708 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 28709 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 28709 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 28780 T64 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 28783 T64 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54488
   [junit4]   2> 28784 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 28784 T64 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 28785 T64 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086
   [junit4]   2> 28785 T64 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/'
   [junit4]   2> 28802 T64 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/solr.xml
   [junit4]   2> 28856 T64 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 28857 T64 oasc.CoreContainer.<init> New CoreContainer 1871905209
   [junit4]   2> 28857 T64 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/]
   [junit4]   2> 28859 T64 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 28860 T64 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 28860 T64 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 28861 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 28861 T64 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 28862 T64 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 28862 T64 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 28863 T64 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 28863 T64 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 28865 T64 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 28866 T64 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 28867 T64 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38544/solr
   [junit4]   2> 28867 T64 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 28868 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28871 T145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48521df3 name:ZooKeeperConnection Watcher:127.0.0.1:38544 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28872 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28876 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28885 T147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e1b71d1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28885 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28892 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29897 T64 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54488_
   [junit4]   2> 29898 T64 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54488_
   [junit4]   2> 29902 T87 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29903 T119 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29903 T105 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29902 T92 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29902 T147 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29904 T133 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 29909 T64 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086
   [junit4]   2> 29910 T64 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/conf/
   [junit4]   2> 29930 T64 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 29931 T64 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 29931 T64 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 29933 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29935 T150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@466054a1 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29935 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29937 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29943 T64 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 29945 T152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46e05be2 name:ZooKeeperConnection Watcher:127.0.0.1:38544/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 29945 T64 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 29947 T64 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 30145 T143 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&maxShardsPerNode=2&name=delLiveColl&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 30149 T87 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 30152 T89 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 30154 T89 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 30856 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 30858 T88 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 30859 T88 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 30875 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T87 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30875 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 30959 T89 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 30960 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:34437_
   [junit4]   2> 30964 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:54488_
   [junit4]   2> 30965 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:36637_
   [junit4]   2> 30965 T89 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:39004_
   [junit4]   2> 30985 T82 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down
   [junit4]   2> 30985 T98 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down
   [junit4]   2> 30985 T140 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down
   [junit4]   2> 30985 T127 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down
   [junit4]   2> 30988 T82 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30989 T98 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30990 T140 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 30990 T127 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 32385 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 32387 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34437",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34437_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32395 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36637",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36637_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32401 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54488",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54488_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32407 T88 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39004",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39004_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 32414 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T87 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32415 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32414 T147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 32989 T82 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 32990 T98 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 32991 T140 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 32991 T127 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 32994 T140 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2
   [junit4]   2> 32994 T98 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1
   [junit4]   2> 32995 T98 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32994 T82 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1
   [junit4]   2> 32994 T127 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2
   [junit4]   2> 32995 T82 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32994 T140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32996 T127 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 32997 T82 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32996 T98 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32997 T82 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32997 T127 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32998 T127 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32997 T140 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:delLiveColl
   [junit4]   2> 32999 T140 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 32998 T98 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 33000 T82 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33001 T140 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33001 T98 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33000 T127 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 33002 T98 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33001 T140 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33001 T82 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33002 T127 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 33007 T98 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T82 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T140 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33008 T127 oasc.ZkController.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 33010 T98 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/'
   [junit4]   2> 33010 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/'
   [junit4]   2> 33011 T140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/'
   [junit4]   2> 33012 T127 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/'
   [junit4]   2> 33102 T127 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33104 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33135 T98 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33159 T140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 33164 T82 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33167 T82 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33190 T82 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 33198 T127 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33200 T127 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33203 T98 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33206 T98 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33231 T127 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 33234 T140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33236 T98 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 33243 T140 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33274 T140 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 33823 T98 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33835 T98 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33839 T82 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33840 T127 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 33847 T82 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33851 T127 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 33852 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33858 T98 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33864 T127 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 33905 T82 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33906 T127 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33911 T98 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33911 T82 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33912 T127 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33918 T127 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33918 T98 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 33920 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33934 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33935 T127 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33931 T82 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33935 T98 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33944 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33944 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33945 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33944 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33946 T82 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33945 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33947 T98 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 33946 T127 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33949 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33950 T82 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33949 T127 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33951 T127 oasc.SolrCore.<init> [delLiveColl_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/
   [junit4]   2> 33950 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 33955 T98 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 33956 T98 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33956 T98 oasc.SolrCore.<init> [delLiveColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/
   [junit4]   2> 33957 T98 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33955 T127 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33950 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 33960 T82 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/
   [junit4]   2> 33960 T82 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 33971 T98 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data
   [junit4]   2> 33972 T98 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index/
   [junit4]   2> 33972 T98 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33973 T98 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty1-1383694732310/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 33974 T98 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33976 T98 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@172be805 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71303650),segFN=segments_1,generation=1}
   [junit4]   2> 33976 T98 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 33979 T127 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data
   [junit4]   2> 33980 T127 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index/
   [junit4]   2> 33980 T127 oasc.SolrCore.initIndex WARN [delLiveColl_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33981 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data
   [junit4]   2> 33981 T127 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteReplicaTest-jetty3-1383694734896/delLiveColl_shard2_replica2/data/index
   [junit4]   2> 33981 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 33982 T127 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33983 T82 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 33984 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-controljetty-1383694731174/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 33985 T82 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 33986 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@143f0608 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@607dcebf),segFN=segments_1,generation=1}
   [junit4]   2> 33986 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 33995 T127 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@399cfbc4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@538f1db2),segFN=segments_1,generation=1}
   [junit4]   2> 33998 T127 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34003 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34004 T140 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 34008 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34012 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34015 T140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34017 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34017 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34018 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34018 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34018 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34019 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34019 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34021 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34020 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34022 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34021 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34023 T127 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34023 T127 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34024 T127 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34022 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34027 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34023 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34027 T127 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34029 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T82 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34030 T98 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34030 T127 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34032 T127 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34033 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34032 T98 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34033 T127 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34035 T98 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34037 T127 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34038 T127 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34038 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34038 T98 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34039 T82 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34041 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34041 T98 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34041 T127 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34041 T98 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34042 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34043 T98 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34043 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34045 T98 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34045 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34046 T98 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34049 T98 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34055 T127 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34055 T98 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34056 T98 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34056 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34056 T127 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34057 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34058 T98 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34058 T140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34059 T127 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34059 T98 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@172be805 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71303650),segFN=segments_1,generation=1}
   [junit4]   2> 34059 T82 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2863346641714839]
   [junit4]   2> 34060 T127 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@399cfbc4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@538f1db2),segFN=segments_1,generation=1}
   [junit4]   2> 34060 T98 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34061 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@143f0608 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@607dcebf),segFN=segments_1,generation=1}
   [junit4]   2> 34061 T127 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34062 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34062 T98 oass.SolrIndexSearcher.<init> Opening Searcher@7de39b29 main
   [junit4]   2> 34063 T82 oass.SolrIndexSearcher.<init> Opening Searcher@a5792d9 main
   [junit4]   2> 34063 T127 oass.SolrIndexSearcher.<init> Opening Searcher@6941817d main
   [junit4]   2> 34074 T140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34080 T140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 34084 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 34084 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 34085 T140 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 34087 T158 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica1] Registered new searcher Searcher@7de39b29 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34091 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 34091 T140 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 34093 T140 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 34094 T140 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/, dataDir=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/
   [junit4]   2> 34102 T140 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a922bc8
   [junit4]   2> 34099 T157 oasc.SolrCore.registerSearcher [delLiveColl_shard2_replica2] Registered new searcher Searcher@6941817d main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34105 T159 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica1] Registered new searcher Searcher@a5792d9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34106 T98 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica1
   [junit4]   2> 34107 T98 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica1 address:http://127.0.0.1:36637 collection:delLiveColl shard:shard2
   [junit4]   2> 34116 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data
   [junit4]   2> 34117 T140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 34117 T140 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory './org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 34118 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteReplicaTest-jetty4-1383694736086/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 34118 T82 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica1
   [junit4]   2> 34119 T140 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21800812862519942]
   [junit4]   2> 34120 T127 oasc.CoreContainer.registerCore registering core: delLiveColl_shard2_replica2
   [junit4]   2> 34119 T82 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica1 address:http://127.0.0.1:34437 collection:delLiveColl shard:shard1
   [junit4]   2> 34120 T127 oasc.ZkController.register Register replica - core:delLiveColl_shard2_replica2 address:http://127.0.0.1:39004 collection:delLiveColl shard:shard2
   [junit4]   2> 34121 T140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c207114 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2a1cf0bb),segFN=segments_1,generation=1}
   [junit4]   2> 34122 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34122 T82 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 34123 T127 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 34132 T98 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard2/election
   [junit4]   2> 34141 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34143 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34144 T127 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 34145 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34145 T82 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 34147 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34149 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34149 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34150 T82 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
   [junit4]   2> 34150 T127 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 34152 T127 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 34152 T140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34153 T140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34153 T140 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34154 T127 oasc.SyncStrategy.sync Sync replica

[...truncated too long message...]

t(AbstractQueuedSynchronizer.java:2052)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]   2> 
   [junit4]   2> 129206 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 129207 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 129706 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 129707 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 130207 T151 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> 
   [junit4]   2> 130707 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 130708 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 131207 T151 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 131208 T151 oaz.ClientCnxn$SendThread.run WARN Session 0x1422aa1fe8a000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 
   [junit4]   2> 110376 T63 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=pt, timezone=America/Detroit
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=2,free=84816760,total=126681088
   [junit4]   2> NOTE: All tests run in this JVM: [TermVectorComponentTest, CircularListTest, SolrCoreTest, UpdateParamsTest, TestMaxScoreQueryParser, DeleteReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.seed=628926A1737E2FE2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt -Dtests.timezone=America/Detroit -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | DeleteReplicaTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:343)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteReplicaTest: 
   [junit4]    >    1) Thread[id=152, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-EventThread, state=WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
   [junit4]    >    2) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:219)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=151, name=TEST-DeleteReplicaTest.testDistribSearch-seed#[628926A1737E2FE2]-SendThread(localhost.localdomain:38544), state=TIMED_WAITING, group=TGRP-DeleteReplicaTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([628926A1737E2FE2]:0)
   [junit4] Completed on J0 in 110.46s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 955 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 334 suites, 1469 tests, 3 suite-level errors, 1 error, 1075 ignored (1 assumption)

Total time: 34 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Thanks Uwe, will revert it and refine the patch. I am still not happy
with how this is implemented but I honestly don't think there is a
cleaner way to run such a "wrapper" target at the top-level of Ant
invocation (including sub-module level).

Dawid

On Tue, Nov 5, 2013 at 11:18 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> it looks like Dawid's commit placed these .tests.totals files in the wrong directory. Should be inside build! Maybe some property is incorrectly initialized.
>
> In addition I will add  another comment to the corresponding issue, because of some dependency of the commit to javascript, that’s not fully guaranteed to be available in every JVM (e.g., on FreeBSD we don't have it).
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
>> Sent: Tuesday, November 05, 2013 10:23 PM
>> To: dev@lucene.apache.org; erick@apache.org; dweiss@apache.org
>> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build #
>> 8189 - Still Failing!
>>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
>> Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 36008 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66:
>> The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-
>> targets.xml:137: Source checkout is dirty after running tests!!! Offending
>> files:
>> * ./lucene/.test-totals-922324336062316929.tmp
>> * ./solr/.test-totals-5710645736889619892.tmp
>>
>> Total time: 50 minutes 15 seconds
>> Build step 'Invoke Ant' marked build as failure Description set: Java:
>> 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
>> Archiving artifacts Recording test results Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

it looks like Dawid's commit placed these .tests.totals files in the wrong directory. Should be inside build! Maybe some property is incorrectly initialized.

In addition I will add  another comment to the corresponding issue, because of some dependency of the commit to javascript, that’s not fully guaranteed to be available in every JVM (e.g., on FreeBSD we don't have it).

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Tuesday, November 05, 2013 10:23 PM
> To: dev@lucene.apache.org; erick@apache.org; dweiss@apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build #
> 8189 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
> Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 36008 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-
> targets.xml:137: Source checkout is dirty after running tests!!! Offending
> files:
> * ./lucene/.test-totals-922324336062316929.tmp
> * ./solr/.test-totals-5710645736889619892.tmp
> 
> Total time: 50 minutes 15 seconds
> Build step 'Invoke Ant' marked build as failure Description set: Java:
> 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
> Archiving artifacts Recording test results Email was triggered for: Failure
> Sending email for trigger: Failure
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8189 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8189/
Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 36008 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:137: Source checkout is dirty after running tests!!! Offending files:
* ./lucene/.test-totals-922324336062316929.tmp
* ./solr/.test-totals-5710645736889619892.tmp

Total time: 50 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b109 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b109) - Build # 8188 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8188/
Java: 64bit/jdk1.8.0-ea-b109 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Tue Nov 05 22:22:36 EAT 2013

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Tue Nov 05 22:22:36 EAT 2013
	at __randomizedtesting.SeedInfo.seed([1DA5EB25240C87F0:EAD6057DE2E42816]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1515)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1264)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 10379 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> 752271 T2946 oas.SolrTestCaseJ4.setUp ###Starting doTestReplicateAfterWrite2Slave
   [junit4]   2> 752275 T2946 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 752282 T2946 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52116
   [junit4]   2> 752282 T2946 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 752282 T2946 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 752283 T2946 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master
   [junit4]   2> 752283 T2946 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/'
   [junit4]   2> 752291 T2946 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/solr.xml
   [junit4]   2> 752299 T2946 oasc.CoreContainer.<init> New CoreContainer 1497975141
   [junit4]   2> 752299 T2946 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/]
   [junit4]   2> 752300 T2946 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 752300 T2946 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 752300 T2946 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 752300 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 752300 T2946 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 752301 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 752301 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 752301 T2946 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 752301 T2946 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 752302 T2946 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 752303 T2946 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 752304 T2956 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1
   [junit4]   2> 752305 T2956 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/'
   [junit4]   2> 752331 T2956 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 752337 T2956 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 752338 T2956 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 752340 T2956 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 752342 T2956 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 752343 T2956 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 752343 T2956 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/data/
   [junit4]   2> 752343 T2956 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 752343 T2956 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/data
   [junit4]   2> 752343 T2956 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/data/index/
   [junit4]   2> 752344 T2956 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 752344 T2956 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352756/master/collection1/data/index
   [junit4]   2> 752344 T2956 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=50.1064453125, floorSegmentMB=1.58203125, forceMergeDeletesPctAllowed=21.402441737506077, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 752345 T2956 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e3da7c1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e38310a),segFN=segments_1,generation=1}
   [junit4]   2> 752345 T2956 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 752345 T2956 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 752346 T2956 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 752348 T2956 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 752348 T2956 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 752348 T2956 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 752349 T2956 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=40.451171875, floorSegmentMB=1.3720703125, forceMergeDeletesPctAllowed=5.764902083071113, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.43676611523563647
   [junit4]   2> 752349 T2956 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e3da7c1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2e38310a),segFN=segments_1,generation=1}
   [junit4]   2> 752349 T2956 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 752349 T2956 oass.SolrIndexSearcher.<init> Opening Searcher@7c267dc3 main
   [junit4]   2> 752350 T2956 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 752350 T2956 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 752350 T2956 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 752350 T2957 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c267dc3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 752351 T2946 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 752351 T2946 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 752354 T2946 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 752356 T2946 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39273
   [junit4]   2> 752356 T2946 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 752357 T2946 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 752357 T2946 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave
   [junit4]   2> 752357 T2946 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/'
   [junit4]   2> 752364 T2946 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/solr.xml
   [junit4]   2> 752381 T2946 oasc.CoreContainer.<init> New CoreContainer 1000197089
   [junit4]   2> 752381 T2946 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/]
   [junit4]   2> 752382 T2946 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 752382 T2946 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 752382 T2946 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 752382 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 752383 T2946 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 752383 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 752383 T2946 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 752383 T2946 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 752383 T2946 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 752384 T2946 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 752384 T2946 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 752385 T2968 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1
   [junit4]   2> 752386 T2968 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/'
   [junit4]   2> 752399 T2968 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 752405 T2968 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 752405 T2968 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 752407 T2968 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 752409 T2968 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 752409 T2968 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 752410 T2968 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/data/
   [junit4]   2> 752410 T2968 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 752410 T2968 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/data
   [junit4]   2> 752410 T2968 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/data/index/
   [junit4]   2> 752410 T2968 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 752411 T2968 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679352836/slave/collection1/data/index
   [junit4]   2> 752411 T2968 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=50.1064453125, floorSegmentMB=1.58203125, forceMergeDeletesPctAllowed=21.402441737506077, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 752412 T2968 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@29693579 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@64656211),segFN=segments_1,generation=1}
   [junit4]   2> 752412 T2968 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 752412 T2968 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 752413 T2968 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 752413 T2968 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 752413 T2968 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 752413 T2968 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 752413 T2968 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 752414 T2968 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 752415 T2968 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 752415 T2968 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 752416 T2968 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 752416 T2968 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=40.451171875, floorSegmentMB=1.3720703125, forceMergeDeletesPctAllowed=5.764902083071113, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.43676611523563647
   [junit4]   2> 752416 T2968 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@29693579 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@64656211),segFN=segments_1,generation=1}
   [junit4]   2> 752416 T2968 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 752416 T2968 oass.SolrIndexSearcher.<init> Opening Searcher@45a80784 main
   [junit4]   2> 752418 T2968 oash.SnapPuller.startExecutorService Poll Scheduled at an interval of 1000ms
   [junit4]   2> 752418 T2969 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45a80784 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 752418 T2968 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 752419 T2946 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 752419 T2946 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C5761 name=collection1 org.apache.solr.core.SolrCore@7c7be323
   [junit4]   2> 752424 T2952 C5761 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=3 
   [junit4]   2> 752427 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 1
   [junit4]   2> 752429 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 0
   [junit4]   2> 752431 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 0
   [junit4]   2> 752434 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
   [junit4]   2> 752436 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 0
   [junit4]   2> 752438 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 0
   [junit4]   2> 752441 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 1
   [junit4]   2> 752443 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 0
   [junit4]   2> 752446 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 1
   [junit4]   2> 752448 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 1
   [junit4]   2> 752450 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 0
   [junit4]   2> 752454 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 1
   [junit4]   2> 752458 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 1
   [junit4]   2> 752460 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 1
   [junit4]   2> 752462 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 1
   [junit4]   2> 752464 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 0
   [junit4]   2> 752466 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 0
   [junit4]   2> 752469 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 1
   [junit4]   2> 752473 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 3
   [junit4]   2> 752476 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 1
   [junit4]   2> 752478 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 1
   [junit4]   2> 752480 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 0
   [junit4]   2> 752482 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 0
   [junit4]   2> 752484 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 0
   [junit4]   2> 752486 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 0
   [junit4]   2> 752488 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 0
   [junit4]   2> 752490 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26]} 0 0
   [junit4]   2> 752493 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27]} 0 1
   [junit4]   2> 752495 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[28]} 0 1
   [junit4]   2> 752497 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[29]} 0 1
   [junit4]   2> 752499 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[30]} 0 1
   [junit4]   2> 752500 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[31]} 0 0
   [junit4]   2> 752503 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[32]} 0 1
   [junit4]   2> 752505 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[33]} 0 1
   [junit4]   2> 752507 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[34]} 0 1
   [junit4]   2> 752509 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[35]} 0 1
   [junit4]   2> 752511 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36]} 0 1
   [junit4]   2> 752513 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[37]} 0 1
   [junit4]   2> 752514 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[38]} 0 0
   [junit4]   2> 752516 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39]} 0 0
   [junit4]   2> 752518 T2953 C5761 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0 
   [junit4]   2> 752518 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[40]} 0 0
   [junit4]   2> 752520 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41]} 0 0
   [junit4]   2> 752522 T2952 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[42]} 0 0
   [junit4]   2> 752524 T2953 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[43]} 0 0
   [junit4]   2> 752526 T2954 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44]} 0 0
   [junit4]   2> 752528 T2955 C5761 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C5762 name=collection1 org.apache.solr.core.SolrCore@7c7be323
   [junit4]   2> 752543 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[46]} 0 1
   [junit4]   2> 752546 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47]} 0 1
   [junit4]   2> 752549 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48]} 0 1
   [junit4]   2> 752551 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[49]} 0 0
   [junit4]   2> 752554 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[50]} 0 1
   [junit4]   2> 752556 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[51]} 0 1
   [junit4]   2> 752558 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[52]} 0 1
   [junit4]   2> 752560 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[53]} 0 1
   [junit4]   2> 752562 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54]} 0 1
   [junit4]   2> 752564 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[55]} 0 1
   [junit4]   2> 752566 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56]} 0 1
   [junit4]   2> 752568 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[57]} 0 1
   [junit4]   2> 752570 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[58]} 0 1
   [junit4]   2> 752572 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[59]} 0 1
   [junit4]   2> 752574 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[60]} 0 1
   [junit4]   2> 752576 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[61]} 0 1
   [junit4]   2> 752578 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[62]} 0 1
   [junit4]   2> 752580 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63]} 0 1
   [junit4]   2> 752582 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64]} 0 1
   [junit4]   2> 752584 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[65]} 0 1
   [junit4]   2> 752586 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66]} 0 1
   [junit4]   2> 752588 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[67]} 0 0
   [junit4]   2> 752590 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[68]} 0 0
   [junit4]   2> 752592 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[69]} 0 0
   [junit4]   2> 752594 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[70]} 0 0
   [junit4]   2> 752596 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[71]} 0 0
   [junit4]   2> 752599 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[72]} 0 1
   [junit4]   2> 752601 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[73]} 0 1
   [junit4]   2> 752603 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[74]} 0 1
   [junit4]   2> 752605 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[75]} 0 1
   [junit4]   2> 752607 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[76]} 0 0
   [junit4]   2> 752609 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[77]} 0 0
   [junit4]   2> 752611 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[78]} 0 0
   [junit4]   2> 752613 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[79]} 0 0
   [junit4]   2> 752615 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[80]} 0 0
   [junit4]   2> 752617 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[81]} 0 0
   [junit4]   2> 752619 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[82]} 0 0
   [junit4]   2> 752621 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[83]} 0 0
   [junit4]   2> 752623 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[84]} 0 0
   [junit4]   2> 752625 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[85]} 0 0
   [junit4]   2> 752627 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[86]} 0 0
   [junit4]   2> 752629 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[87]} 0 0
   [junit4]   2> 752631 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[88]} 0 0
   [junit4]   2> 752633 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[89]} 0 0
   [junit4]   2> 752635 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[90]} 0 0
   [junit4]   2> 752637 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[91]} 0 0
   [junit4]   2> 752639 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[92]} 0 0
   [junit4]   2> 752641 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[93]} 0 0
   [junit4]   2> 752643 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[94]} 0 0
   [junit4]   2> 752645 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[95]} 0 0
   [junit4]   2> 752647 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[96]} 0 0
   [junit4]   2> 752649 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[97]} 0 0
   [junit4]   2> 752651 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[98]} 0 0
   [junit4]   2> 752653 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[99]} 0 0
   [junit4]   2> 752655 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[100]} 0 0
   [junit4]   2> 752657 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[101]} 0 0
   [junit4]   2> 752659 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[102]} 0 0
   [junit4]   2> 752661 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[103]} 0 0
   [junit4]   2> 752663 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[104]} 0 0
   [junit4]   2> 752665 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[105]} 0 0
   [junit4]   2> 752667 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[106]} 0 0
   [junit4]   2> 752669 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[107]} 0 0
   [junit4]   2> 752671 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[108]} 0 0
   [junit4]   2> 752673 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[109]} 0 0
   [junit4]   2> 752675 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[110]} 0 0
   [junit4]   2> 752677 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[111]} 0 0
   [junit4]   2> 752679 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[112]} 0 0
   [junit4]   2> 752681 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[113]} 0 0
   [junit4]   2> 752683 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[114]} 0 0
   [junit4]   2> 752685 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[115]} 0 0
   [junit4]   2> 752687 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[116]} 0 0
   [junit4]   2> 752690 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[117]} 0 1
   [junit4]   2> 752692 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[118]} 0 1
   [junit4]   2> 752694 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[119]} 0 1
   [junit4]   2> 752696 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[120]} 0 0
   [junit4]   2> 752698 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[121]} 0 0
   [junit4]   2> 752700 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[122]} 0 0
   [junit4]   2> 752702 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[123]} 0 0
   [junit4]   2> 752704 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[124]} 0 0
   [junit4]   2> 752706 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[125]} 0 0
   [junit4]   2> 752708 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[126]} 0 0
   [junit4]   2> 752710 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[127]} 0 0
   [junit4]   2> 752712 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[128]} 0 0
   [junit4]   2> 752714 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[129]} 0 0
   [junit4]   2> 752716 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[130]} 0 0
   [junit4]   2> 752718 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[131]} 0 0
   [junit4]   2> 752721 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[132]} 0 1
   [junit4]   2> 752723 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[133]} 0 1
   [junit4]   2> 752725 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[134]} 0 1
   [junit4]   2> 752727 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[135]} 0 1
   [junit4]   2> 752729 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[136]} 0 1
   [junit4]   2> 752731 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[137]} 0 1
   [junit4]   2> 752733 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[138]} 0 0
   [junit4]   2> 752735 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[139]} 0 1
   [junit4]   2> 752737 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[140]} 0 1
   [junit4]   2> 752739 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[141]} 0 1
   [junit4]   2> 752740 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[142]} 0 0
   [junit4]   2> 752742 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[143]} 0 0
   [junit4]   2> 752745 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[144]} 0 1
   [junit4]   2> 752748 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[145]} 0 1
   [junit4]   2> 752750 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[146]} 0 0
   [junit4]   2> 752753 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[147]} 0 0
   [junit4]   2> 752756 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[148]} 0 1
   [junit4]   2> 752759 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[149]} 0 1
   [junit4]   2> 752762 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[150]} 0 1
   [junit4]   2> 752764 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[151]} 0 0
   [junit4]   2> 752767 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[152]} 0 1
   [junit4]   2> 752770 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[153]} 0 1
   [junit4]   2> 752773 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[154]} 0 1
   [junit4]   2> 752775 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[155]} 0 0
   [junit4]   2> 752777 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[156]} 0 1
   [junit4]   2> 752779 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[157]} 0 1
   [junit4]   2> 752781 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[158]} 0 1
   [junit4]   2> 752783 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[159]} 0 1
   [junit4]   2> 752785 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[160]} 0 1
   [junit4]   2> 752787 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[161]} 0 1
   [junit4]   2> 752788 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[162]} 0 0
   [junit4]   2> 752790 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[163]} 0 0
   [junit4]   2> 752792 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[164]} 0 0
   [junit4]   2> 752794 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[165]} 0 0
   [junit4]   2> 752796 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[166]} 0 0
   [junit4]   2> 752798 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[167]} 0 0
   [junit4]   2> 752800 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[168]} 0 0
   [junit4]   2> 752802 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[169]} 0 0
   [junit4]   2> 752805 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[170]} 0 1
   [junit4]   2> 752808 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[171]} 0 1
   [junit4]   2> 752811 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[172]} 0 1
   [junit4]   2> 752813 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[173]} 0 0
   [junit4]   2> 752815 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[174]} 0 0
   [junit4]   2> 752817 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[175]} 0 0
   [junit4]   2> 752819 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[176]} 0 0
   [junit4]   2> 752822 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[177]} 0 1
   [junit4]   2> 752824 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[178]} 0 1
   [junit4]   2> 752826 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[179]} 0 1
   [junit4]   2> 752828 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[180]} 0 1
   [junit4]   2> 752830 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[181]} 0 1
   [junit4]   2> 752832 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[182]} 0 1
   [junit4]   2> 752834 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[183]} 0 1
   [junit4]   2> 752836 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[184]} 0 1
   [junit4]   2> 752838 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[185]} 0 1
   [junit4]   2> 752840 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[186]} 0 1
   [junit4]   2> 752842 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[187]} 0 1
   [junit4]   2> 752844 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[188]} 0 1
   [junit4]   2> 752846 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[189]} 0 1
   [junit4]   2> 752848 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[190]} 0 1
   [junit4]   2> 752850 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[191]} 0 1
   [junit4]   2> 752852 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[192]} 0 1
   [junit4]   2> 752854 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[193]} 0 1
   [junit4]   2> 752855 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[194]} 0 0
   [junit4]   2> 752857 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[195]} 0 0
   [junit4]   2> 752859 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[196]} 0 0
   [junit4]   2> 752861 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[197]} 0 0
   [junit4]   2> 752863 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[198]} 0 0
   [junit4]   2> 752865 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[199]} 0 0
   [junit4]   2> 752867 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[200]} 0 0
   [junit4]   2> 752869 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[201]} 0 0
   [junit4]   2> 752871 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[202]} 0 0
   [junit4]   2> 752873 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[203]} 0 0
   [junit4]   2> 752875 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[204]} 0 1
   [junit4]   2> 752877 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[205]} 0 1
   [junit4]   2> 752879 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[206]} 0 1
   [junit4]   2> 752880 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[207]} 0 0
   [junit4]   2> 752882 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[208]} 0 0
   [junit4]   2> 752884 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[209]} 0 0
   [junit4]   2> 752886 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[210]} 0 0
   [junit4]   2> 752888 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[211]} 0 0
   [junit4]   2> 752890 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[212]} 0 0
   [junit4]   2> 752892 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[213]} 0 0
   [junit4]   2> 752894 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[214]} 0 0
   [junit4]   2> 752896 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[215]} 0 1
   [junit4]   2> 752898 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[216]} 0 1
   [junit4]   2> 752900 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[217]} 0 1
   [junit4]   2> 752902 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[218]} 0 1
   [junit4]   2> 752904 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[219]} 0 1
   [junit4]   2> 752907 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[220]} 0 1
   [junit4]   2> 752910 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[221]} 0 1
   [junit4]   2> 752912 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[222]} 0 0
   [junit4]   2> 752915 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[223]} 0 1
   [junit4]   2> 752917 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[224]} 0 1
   [junit4]   2> 752919 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[225]} 0 1
   [junit4]   2> 752920 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[226]} 0 0
   [junit4]   2> 752922 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[227]} 0 0
   [junit4]   2> 752924 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[228]} 0 0
   [junit4]   2> 752926 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[229]} 0 0
   [junit4]   2> 752928 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[230]} 0 0
   [junit4]   2> 752930 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[231]} 0 1
   [junit4]   2> 752932 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[232]} 0 1
   [junit4]   2> 752934 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[233]} 0 0
   [junit4]   2> 752936 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[234]} 0 0
   [junit4]   2> 752938 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[235]} 0 0
   [junit4]   2> 752940 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[236]} 0 0
   [junit4]   2> 752942 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[237]} 0 0
   [junit4]   2> 752944 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[238]} 0 0
   [junit4]   2> 752946 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[239]} 0 0
   [junit4]   2> 752948 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[240]} 0 0
   [junit4]   2> 752950 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[241]} 0 0
   [junit4]   2> 752952 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[242]} 0 0
   [junit4]   2> 752954 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[243]} 0 0
   [junit4]   2> 752956 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[244]} 0 0
   [junit4]   2> 752958 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[245]} 0 0
   [junit4]   2> 752960 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[246]} 0 0
   [junit4]   2> 752962 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[247]} 0 0
   [junit4]   2> 752964 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[248]} 0 0
   [junit4]   2> 752966 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[249]} 0 1
   [junit4]   2> 752968 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[250]} 0 1
   [junit4]   2> 752970 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[251]} 0 1
   [junit4]   2> 752972 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[252]} 0 1
   [junit4]   2> 752974 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[253]} 0 1
   [junit4]   2> 752975 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[254]} 0 0
   [junit4]   2> 752977 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[255]} 0 0
   [junit4]   2> 752979 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[256]} 0 0
   [junit4]   2> 752981 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[257]} 0 0
   [junit4]   2> 752983 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[258]} 0 0
   [junit4]   2> 752985 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[259]} 0 0
   [junit4]   2> 752987 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[260]} 0 0
   [junit4]   2> 752989 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[261]} 0 0
   [junit4]   2> 752991 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[262]} 0 0
   [junit4]   2> 752993 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[263]} 0 0
   [junit4]   2> 752995 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[264]} 0 0
   [junit4]   2> 752997 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[265]} 0 0
   [junit4]   2> 753000 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[266]} 0 1
   [junit4]   2> 753003 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[267]} 0 1
   [junit4]   2> 753006 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[268]} 0 1
   [junit4]   2> 753008 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[269]} 0 0
   [junit4]   2> 753011 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[270]} 0 1
   [junit4]   2> 753014 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[271]} 0 0
   [junit4]   2> 753016 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[272]} 0 1
   [junit4]   2> 753018 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[273]} 0 1
   [junit4]   2> 753020 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[274]} 0 1
   [junit4]   2> 753022 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[275]} 0 1
   [junit4]   2> 753024 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[276]} 0 1
   [junit4]   2> 753026 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[277]} 0 1
   [junit4]   2> 753028 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[278]} 0 1
   [junit4]   2> 753030 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[279]} 0 1
   [junit4]   2> 753032 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[280]} 0 1
   [junit4]   2> 753034 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[281]} 0 1
   [junit4]   2> 753036 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[282]} 0 1
   [junit4]   2> 753038 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[283]} 0 1
   [junit4]   2> 753040 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[284]} 0 1
   [junit4]   2> 753041 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[285]} 0 0
   [junit4]   2> 753044 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[286]} 0 1
   [junit4]   2> 753046 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[287]} 0 1
   [junit4]   2> 753048 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[288]} 0 1
   [junit4]   2> 753049 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[289]} 0 0
   [junit4]   2> 753052 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[290]} 0 1
   [junit4]   2> 753054 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[291]} 0 1
   [junit4]   2> 753056 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[292]} 0 1
   [junit4]   2> 753058 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[293]} 0 1
   [junit4]   2> 753060 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[294]} 0 1
   [junit4]   2> 753062 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[295]} 0 1
   [junit4]   2> 753064 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[296]} 0 1
   [junit4]   2> 753066 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[297]} 0 1
   [junit4]   2> 753068 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[298]} 0 1
   [junit4]   2> 753070 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[299]} 0 0
   [junit4]   2> 753072 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[300]} 0 0
   [junit4]   2> 753074 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[301]} 0 1
   [junit4]   2> 753076 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[302]} 0 1
   [junit4]   2> 753078 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[303]} 0 0
   [junit4]   2> 753080 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[304]} 0 0
   [junit4]   2> 753082 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[305]} 0 1
   [junit4]   2> 753084 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[306]} 0 0
   [junit4]   2> 753086 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[307]} 0 0
   [junit4]   2> 753088 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[308]} 0 0
   [junit4]   2> 753090 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[309]} 0 0
   [junit4]   2> 753093 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[310]} 0 1
   [junit4]   2> 753095 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[311]} 0 1
   [junit4]   2> 753097 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[312]} 0 1
   [junit4]   2> 753099 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[313]} 0 1
   [junit4]   2> 753101 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[314]} 0 1
   [junit4]   2> 753103 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[315]} 0 1
   [junit4]   2> 753105 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[316]} 0 1
   [junit4]   2> 753107 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[317]} 0 1
   [junit4]   2> 753109 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[318]} 0 0
   [junit4]   2> 753111 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[319]} 0 1
   [junit4]   2> 753113 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[320]} 0 1
   [junit4]   2> 753115 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[321]} 0 1
   [junit4]   2> 753117 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[322]} 0 0
   [junit4]   2> 753119 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[323]} 0 0
   [junit4]   2> 753121 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[324]} 0 0
   [junit4]   2> 753123 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[325]} 0 0
   [junit4]   2> 753125 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[326]} 0 0
   [junit4]   2> 753127 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[327]} 0 0
   [junit4]   2> 753129 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[328]} 0 0
   [junit4]   2> 753131 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[329]} 0 0
   [junit4]   2> 753133 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[330]} 0 0
   [junit4]   2> 753135 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[331]} 0 0
   [junit4]   2> 753137 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[332]} 0 0
   [junit4]   2> 753139 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[333]} 0 0
   [junit4]   2> 753142 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[334]} 0 1
   [junit4]   2> 753144 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[335]} 0 1
   [junit4]   2> 753145 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[336]} 0 0
   [junit4]   2> 753148 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[337]} 0 1
   [junit4]   2> 753150 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[338]} 0 1
   [junit4]   2> 753152 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[339]} 0 1
   [junit4]   2> 753154 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[340]} 0 1
   [junit4]   2> 753156 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[341]} 0 1
   [junit4]   2> 753158 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[342]} 0 1
   [junit4]   2> 753160 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[343]} 0 1
   [junit4]   2> 753162 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[344]} 0 1
   [junit4]   2> 753164 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[345]} 0 1
   [junit4]   2> 753166 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[346]} 0 1
   [junit4]   2> 753168 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[347]} 0 1
   [junit4]   2> 753169 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[348]} 0 0
   [junit4]   2> 753172 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[349]} 0 1
   [junit4]   2> 753174 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[350]} 0 1
   [junit4]   2> 753176 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[351]} 0 1
   [junit4]   2> 753178 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[352]} 0 1
   [junit4]   2> 753180 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[353]} 0 1
   [junit4]   2> 753182 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[354]} 0 0
   [junit4]   2> 753184 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[355]} 0 0
   [junit4]   2> 753186 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[356]} 0 0
   [junit4]   2> 753188 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[357]} 0 0
   [junit4]   2> 753190 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[358]} 0 0
   [junit4]   2> 753192 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[359]} 0 0
   [junit4]   2> 753194 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[360]} 0 0
   [junit4]   2> 753196 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[361]} 0 0
   [junit4]   2> 753198 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[362]} 0 0
   [junit4]   2> 753200 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[363]} 0 0
   [junit4]   2> 753202 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[364]} 0 0
   [junit4]   2> 753204 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[365]} 0 0
   [junit4]   2> 753207 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[366]} 0 1
   [junit4]   2> 753208 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[367]} 0 0
   [junit4]   2> 753210 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[368]} 0 0
   [junit4]   2> 753213 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[369]} 0 1
   [junit4]   2> 753215 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[370]} 0 1
   [junit4]   2> 753217 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[371]} 0 1
   [junit4]   2> 753219 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[372]} 0 1
   [junit4]   2> 753221 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[373]} 0 1
   [junit4]   2> 753223 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[374]} 0 1
   [junit4]   2> 753225 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[375]} 0 1
   [junit4]   2> 753227 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[376]} 0 1
   [junit4]   2> 753229 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[377]} 0 0
   [junit4]   2> 753231 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[378]} 0 0
   [junit4]   2> 753234 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[379]} 0 1
   [junit4]   2> 753236 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[380]} 0 1
   [junit4]   2> 753238 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[381]} 0 1
   [junit4]   2> 753239 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[382]} 0 0
   [junit4]   2> 753241 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[383]} 0 0
   [junit4]   2> 753243 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[384]} 0 0
   [junit4]   2> 753245 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[385]} 0 0
   [junit4]   2> 753247 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[386]} 0 0
   [junit4]   2> 753249 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[387]} 0 0
   [junit4]   2> 753251 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[388]} 0 0
   [junit4]   2> 753253 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[389]} 0 0
   [junit4]   2> 753255 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[390]} 0 1
   [junit4]   2> 753257 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[391]} 0 1
   [junit4]   2> 753259 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[392]} 0 1
   [junit4]   2> 753261 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[393]} 0 1
   [junit4]   2> 753263 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[394]} 0 1
   [junit4]   2> 753264 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[395]} 0 0
   [junit4]   2> 753266 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[396]} 0 0
   [junit4]   2> 753268 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[397]} 0 0
   [junit4]   2> 753270 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[398]} 0 0
   [junit4]   2> 753272 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[399]} 0 0
   [junit4]   2> 753274 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[400]} 0 0
   [junit4]   2> 753276 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[401]} 0 0
   [junit4]   2> 753278 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[402]} 0 0
   [junit4]   2> 753280 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[403]} 0 1
   [junit4]   2> 753282 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[404]} 0 1
   [junit4]   2> 753284 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[405]} 0 1
   [junit4]   2> 753286 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[406]} 0 0
   [junit4]   2> 753288 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[407]} 0 0
   [junit4]   2> 753290 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[408]} 0 0
   [junit4]   2> 753292 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[409]} 0 0
   [junit4]   2> 753294 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[410]} 0 0
   [junit4]   2> 753296 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[411]} 0 0
   [junit4]   2> 753298 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[412]} 0 0
   [junit4]   2> 753300 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[413]} 0 0
   [junit4]   2> 753302 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[414]} 0 0
   [junit4]   2> 753304 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[415]} 0 0
   [junit4]   2> 753306 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[416]} 0 0
   [junit4]   2> 753309 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[417]} 0 1
   [junit4]   2> 753311 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[418]} 0 1
   [junit4]   2> 753313 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[419]} 0 1
   [junit4]   2> 753315 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[420]} 0 1
   [junit4]   2> 753317 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[421]} 0 1
   [junit4]   2> 753319 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[422]} 0 1
   [junit4]   2> 753321 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[423]} 0 1
   [junit4]   2> 753323 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[424]} 0 1
   [junit4]   2> 753325 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[425]} 0 1
   [junit4]   2> 753326 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[426]} 0 0
   [junit4]   2> 753328 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[427]} 0 0
   [junit4]   2> 753331 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[428]} 0 1
   [junit4]   2> 753333 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[429]} 0 1
   [junit4]   2> 753335 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[430]} 0 1
   [junit4]   2> 753336 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[431]} 0 0
   [junit4]   2> 753338 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[432]} 0 0
   [junit4]   2> 753341 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[433]} 0 1
   [junit4]   2> 753343 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[434]} 0 1
   [junit4]   2> 753345 T2952 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[435]} 0 1
   [junit4]   2> 753347 T2953 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[436]} 0 1
   [junit4]   2> 753349 T2954 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[437]} 0 1
   [junit4]   2> 753350 T2955 C5762 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=ja

[...truncated too long message...]

ctory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data/index.20131105222330016
   [junit4]   2> ASYNC  NEW_CORE C5824 name=collection1 org.apache.solr.core.SolrCore@4eae8ff2
   [junit4]   2> 809557 T3556 C5824 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=492 status=0 QTime=0 
   [junit4]   2> 809558 T2946 oas.SolrTestCaseJ4.tearDown ###Ending doTestReplicateAfterStartup
   [junit4]   2> 809579 T2946 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1102998594
   [junit4]   2> 809580 T2946 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4f255e2f
   [junit4]   2> 809580 T2946 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=492,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 809581 T2946 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 809581 T2946 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 809581 T2946 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 809582 T2946 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 809582 T2946 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 809582 T2946 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data/index;done=false>>]
   [junit4]   2> 809583 T2946 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data/index
   [junit4]   2> 809583 T2946 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data;done=false>>]
   [junit4]   2> 809583 T2946 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407529/master/collection1/data
   [junit4]   2> 809583 T2946 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 809655 T2946 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=105098583
   [junit4]   2> 809656 T2946 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4eae8ff2
   [junit4]   2> 809656 T2946 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 809657 T2946 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 809657 T2946 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 809657 T2946 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 809658 T2946 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 809658 T2946 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 809658 T2946 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data/index;done=false>>]
   [junit4]   2> 809659 T2946 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data/index
   [junit4]   2> 809659 T2946 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data;done=false>>]
   [junit4]   2> 809659 T2946 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1383679407609/slave/collection1/data
   [junit4]   2> 809659 T2946 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 809717 T2946 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=ru_RU, timezone=Africa/Kampala
   [junit4]   2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=68962136,total=244363264
   [junit4]   2> NOTE: All tests run in this JVM: [TestImplicitCoreProperties, AlternateDirectoryTest, TestCollationField, DocumentAnalysisRequestHandlerTest, ChangedSchemaMergeTest, TestDefaultSimilarityFactory, SortByFunctionTest, TestUpdate, TestZkChroot, SignatureUpdateProcessorFactoryTest, PluginInfoTest, RAMDirectoryFactoryTest, MinimalSchemaTest, TestAnalyzeInfixSuggestions, TestIndexingPerformance, HighlighterConfigTest, TestFieldResource, TestCloudManagedSchemaAddField, XsltUpdateRequestHandlerTest, TestStressReorder, TestArbitraryIndexDir, XmlUpdateRequestHandlerTest, NoCacheHeaderTest, TestRandomFaceting, TestJmxIntegration, DebugComponentTest, TestStressRecovery, TestCodecSupport, ZkNodePropsTest, TestAnalyzedSuggestions, TestFastWriter, TestSolrJ, AddSchemaFieldsUpdateProcessorFactoryTest, TestXIncludeConfig, TestComponentsName, TestSchemaSimilarityResource, TestIBSimilarityFactory, RequestHandlersTest, DocValuesMultiTest, TestDynamicFieldResource, TestFoldingMultitermQuery, TimeZoneUtilsTest, SolrTestCaseJ4Test, UniqFieldsUpdateProcessorFactoryTest, DefaultValueUpdateProcessorTest, DeleteShardTest, NotRequiredUniqueKeyTest, HighlighterTest, TestBinaryResponseWriter, RequiredFieldsTest, FullSolrCloudDistribCmdsTest, CoreMergeIndexesAdminHandlerTest, HdfsSyncSliceTest, ConvertedLegacyTest, OutputWriterTest, PathHierarchyTokenizerFactoryTest, TestStressLucene, TestRangeQuery, CurrencyFieldXmlFileTest, HardAutoCommitTest, DistanceFunctionTest, DirectSolrConnectionTest, TestPhraseSuggestions, BadIndexSchemaTest, TestHashPartitioner, PingRequestHandlerTest, TestBinaryField, TestFuzzyAnalyzedSuggestions, TestDFRSimilarityFactory, TestWordDelimiterFilterFactory, SolrCmdDistributorTest, AliasIntegrationTest, TestSolrDeletionPolicy2, IndexSchemaRuntimeFieldTest, TestAddFieldRealTimeGet, TestCSVResponseWriter, CachingDirectoryFactoryTest, TestMaxScoreQueryParser, TestQuerySenderNoQuery, SolrRequestParserTest, TestMultiCoreConfBootstrap, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, TestReplicationHandler]
   [junit4] Completed on J1 in 57.46s, 14 tests, 1 failure, 1 skipped <<< FAILURES!

[...truncated 476 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 333 suites, 1463 tests, 1 failure, 41 ignored (6 assumptions)

Total time: 36 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b109 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure