You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/22 18:33:53 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_55) - Build # 10240 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10240/
Java: 32bit/jdk1.7.0_55 -client -XX:+UseSerialGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
ERROR: SolrZkClient opens=17 closes=16

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=17 closes=16
	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
	at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:     1) Thread[id=1905, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    2) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=1905, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   2) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
deletecollection the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: deletecollection the collection time out:180s
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:850)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11267 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/init-core-data-001
   [junit4]   2> 792440 T1749 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 792441 T1749 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 792444 T1749 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 792444 T1749 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 792445 T1750 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 792545 T1749 oasc.ZkTestServer.run start zk server on port:56904
   [junit4]   2> 792546 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 793665 T1753 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1118ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 793668 T1756 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16e844c name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 793668 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 793669 T1749 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 793672 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 793673 T1758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c016f name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 793673 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 793673 T1749 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 793675 T1749 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 793677 T1749 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 793678 T1749 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 793680 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 793680 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 793683 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 793683 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 793689 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 793689 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 793691 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 793692 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 793693 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 793694 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 793696 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 793696 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 793699 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 793699 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 793701 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 793702 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 793704 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 793704 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 793707 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 793708 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 793710 T1749 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 793710 T1749 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 793967 T1749 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 793971 T1749 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58694
   [junit4]   2> 793971 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 793972 T1749 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 793972 T1749 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002
   [junit4]   2> 793972 T1749 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/'
   [junit4]   2> 793986 T1749 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solr.xml
   [junit4]   2> 794030 T1749 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002
   [junit4]   2> 794031 T1749 oasc.CoreContainer.<init> New CoreContainer 16835787
   [junit4]   2> 794031 T1749 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/]
   [junit4]   2> 794032 T1749 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 794033 T1749 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 794033 T1749 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 794033 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 794034 T1749 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 794034 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 794034 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 794035 T1749 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 794035 T1749 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 794035 T1749 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 794036 T1749 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 794036 T1749 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 794036 T1749 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56904/solr
   [junit4]   2> 794037 T1749 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 794037 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 794046 T1769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ace595 name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 794047 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 794049 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 794050 T1771 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f402a name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 794051 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 794052 T1749 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 794054 T1749 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 794056 T1749 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 794058 T1749 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 794060 T1749 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 794071 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 794074 T1749 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58694_
   [junit4]   2> 794075 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58694_
   [junit4]   2> 794078 T1749 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 794080 T1749 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 794083 T1749 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58694_
   [junit4]   2> 794083 T1749 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 794087 T1749 oasc.Overseer.start Overseer (id=91801260541280259-127.0.0.1:58694_-n_0000000000) starting
   [junit4]   2> 794090 T1749 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 794097 T1773 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 794097 T1749 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 794100 T1749 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 794101 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 794103 T1772 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 794106 T1749 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002
   [junit4]   2> 794112 T1749 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/conf/
   [junit4]   2> 794213 T1749 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 794214 T1749 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 794215 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 794216 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 794217 T1776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8966d name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 794217 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 794218 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 794220 T1749 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 794464 T1749 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 794465 T1749 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 794466 T1749 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55531
   [junit4]   2> 794467 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 794468 T1749 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 794468 T1749 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003
   [junit4]   2> 794468 T1749 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/'
   [junit4]   2> 794481 T1749 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solr.xml
   [junit4]   2> 794533 T1749 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003
   [junit4]   2> 794534 T1749 oasc.CoreContainer.<init> New CoreContainer 11626707
   [junit4]   2> 794534 T1749 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/]
   [junit4]   2> 794535 T1749 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 794535 T1749 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 794536 T1749 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 794536 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 794536 T1749 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 794536 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 794537 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 794537 T1749 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 794537 T1749 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 794538 T1749 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 794538 T1749 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 794539 T1749 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 794539 T1749 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56904/solr
   [junit4]   2> 794539 T1749 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 794540 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 794541 T1787 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@158fb27 name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 794542 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 794545 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 794546 T1789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bdbf88 name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 794546 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 794552 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 795553 T1749 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55531_
   [junit4]   2> 795555 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55531_
   [junit4]   2> 795558 T1776 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 795558 T1789 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 795558 T1771 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 795562 T1749 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003
   [junit4]   2> 795568 T1749 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/conf/
   [junit4]   2> 795676 T1749 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 795677 T1749 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 795678 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 795948 T1749 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 795949 T1749 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 795951 T1749 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37426
   [junit4]   2> 795952 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 795952 T1749 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 795952 T1749 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004
   [junit4]   2> 795953 T1749 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004/'
   [junit4]   2> 795966 T1749 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004/solr.xml
   [junit4]   2> 796011 T1749 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004
   [junit4]   2> 796012 T1749 oasc.CoreContainer.<init> New CoreContainer 3851876
   [junit4]   2> 796012 T1749 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004/]
   [junit4]   2> 796013 T1749 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 796013 T1749 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 796014 T1749 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 796014 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 796014 T1749 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 796015 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 796015 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 796015 T1749 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 796015 T1749 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 796016 T1749 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 796016 T1749 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 796017 T1749 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 796017 T1749 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56904/solr
   [junit4]   2> 796017 T1749 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 796018 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 796019 T1801 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b6e3de name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 796020 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 796021 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 796022 T1803 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1861ecb name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 796022 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 796027 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 797029 T1749 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37426_
   [junit4]   2> 797030 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37426_
   [junit4]   2> 797032 T1789 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 797032 T1776 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 797032 T1803 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 797032 T1771 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 797036 T1749 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004
   [junit4]   2> 797042 T1749 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004/conf/
   [junit4]   2> 797140 T1749 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 797142 T1749 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 797142 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 797395 T1749 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 797395 T1749 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 797398 T1749 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46138
   [junit4]   2> 797399 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 797400 T1749 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 797400 T1749 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005
   [junit4]   2> 797401 T1749 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/'
   [junit4]   2> 797428 T1749 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solr.xml
   [junit4]   2> 797473 T1749 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005
   [junit4]   2> 797474 T1749 oasc.CoreContainer.<init> New CoreContainer 18893250
   [junit4]   2> 797474 T1749 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/]
   [junit4]   2> 797475 T1749 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 797476 T1749 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 797476 T1749 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 797476 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 797477 T1749 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 797477 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 797477 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 797478 T1749 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 797478 T1749 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 797478 T1749 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 797479 T1749 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 797479 T1749 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 797479 T1749 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56904/solr
   [junit4]   2> 797480 T1749 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 797480 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 797482 T1815 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca6d77 name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 797482 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 797484 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 797485 T1817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@622769 name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 797486 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 797490 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 798493 T1749 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46138_
   [junit4]   2> 798494 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46138_
   [junit4]   2> 798496 T1817 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 798496 T1803 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 798496 T1771 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 798496 T1789 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 798496 T1776 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 798510 T1749 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005
   [junit4]   2> 798521 T1749 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/conf/
   [junit4]   2> 798666 T1749 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 798667 T1749 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 798668 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 798942 T1749 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 798943 T1749 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 798945 T1749 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53777
   [junit4]   2> 798945 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 798946 T1749 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 798946 T1749 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006
   [junit4]   2> 798946 T1749 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006/'
   [junit4]   2> 798960 T1749 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006/solr.xml
   [junit4]   2> 799017 T1749 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006
   [junit4]   2> 799018 T1749 oasc.CoreContainer.<init> New CoreContainer 27500342
   [junit4]   2> 799018 T1749 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006/]
   [junit4]   2> 799019 T1749 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 799020 T1749 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 799020 T1749 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 799020 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 799020 T1749 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 799021 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 799021 T1749 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 799021 T1749 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 799022 T1749 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 799022 T1749 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 799023 T1749 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 799023 T1749 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 799023 T1749 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56904/solr
   [junit4]   2> 799023 T1749 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 799024 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 799025 T1829 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129b905 name:ZooKeeperConnection Watcher:127.0.0.1:56904 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 799026 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 799028 T1749 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 799029 T1831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a640be name:ZooKeeperConnection Watcher:127.0.0.1:56904/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 799030 T1749 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 799039 T1749 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 800041 T1749 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53777_
   [junit4]   2> 800044 T1749 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53777_
   [junit4]   2> 800049 T1776 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800049 T1817 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800050 T1831 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800050 T1803 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800050 T1771 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800049 T1789 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 800058 T1749 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006
   [junit4]   2> 800069 T1749 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-006/conf/
   [junit4]   2> 800189 T1749 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 800190 T1749 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 800191 T1749 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 800194 T1782 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&action=CREATE&replicationFactor=2&name=solrj_collection&version=2&collection.configName=conf1&numShards=2&wt=javabin&router.name=compositeId
   [junit4]   2> 800195 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800196 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800196 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800196 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800197 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800197 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800197 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800199 T1773 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 800200 T1833 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 800201 T1833 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 800202 T1833 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 800207 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800210 T1772 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 800211 T1772 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 800211 T1772 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 800215 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800216 T1817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800216 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800216 T1771 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800216 T1803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800216 T1789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800216 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800308 T1833 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 800308 T1833 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:58694_
   [junit4]   2> 800310 T1833 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:46138_
   [junit4]   2> 800312 T1833 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:37426_
   [junit4]   2> 800313 T1766 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&name=solrj_collection_shard1_replica1&version=2&collection=solrj_collection&collection.configName=conf1&shard=shard1&numShards=2&wt=javabin
   [junit4]   2> 800314 T1810 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&name=solrj_collection_shard1_replica2&version=2&collection=solrj_collection&collection.configName=conf1&shard=shard1&numShards=2&wt=javabin
   [junit4]   2> 800315 T1833 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:55531_
   [junit4]   2> 800316 T1766 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 800316 T1798 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&name=solrj_collection_shard2_replica1&version=2&collection=solrj_collection&collection.configName=conf1&shard=shard2&numShards=2&wt=javabin
   [junit4]   2> 800317 T1810 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 800318 T1766 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 800318 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800319 T1810 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 800319 T1798 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 800318 T1783 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&name=solrj_collection_shard2_replica2&version=2&collection=solrj_collection&collection.configName=conf1&shard=shard2&numShards=2&wt=javabin
   [junit4]   2> 800321 T1772 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 800322 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800322 T1798 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 800323 T1772 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58694",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58694_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 800323 T1783 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 800326 T1783 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 800328 T1772 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46138",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46138_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 800332 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800333 T1772 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37426",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37426_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 800336 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800338 T1772 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55531",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55531_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 800341 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 800443 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800443 T1771 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800443 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800443 T1817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800443 T1803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 800443 T1789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 801319 T1766 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 801320 T1810 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 801320 T1766 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 801321 T1810 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 801321 T1766 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 801322 T1810 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 801322 T1766 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 801322 T1810 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 801323 T1798 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 801323 T1810 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 801323 T1766 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 801324 T1798 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 801324 T1810 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/'
   [junit4]   2> 801325 T1798 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 801325 T1798 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 801324 T1766 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/'
   [junit4]   2> 801326 T1798 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 801326 T1783 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 801327 T1798 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-004/solrj_collection_shard2_replica1/'
   [junit4]   2> 801327 T1783 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 801329 T1783 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 801329 T1783 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 801330 T1783 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 801330 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/'
   [junit4]   2> 801390 T1783 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 801409 T1810 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 801416 T1766 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 801428 T1798 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 801486 T1810 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 801487 T1810 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 801494 T1766 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 801495 T1766 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 801495 T1783 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 801496 T1810 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 801497 T1783 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 801505 T1766 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 801511 T1783 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 801535 T1798 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 801536 T1798 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 801546 T1798 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 802094 T1810 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802095 T1810 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802096 T1810 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 802103 T1810 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 802106 T1810 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 802120 T1810 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802124 T1810 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802128 T1810 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802129 T1810 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802130 T1810 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802130 T1810 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802131 T1810 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802132 T1810 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802132 T1810 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 802132 T1810 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 802133 T1810 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/data/
   [junit4]   2> 802133 T1810 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a2e5c6
   [junit4]   2> 802135 T1810 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/data
   [junit4]   2> 802135 T1810 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 802135 T1810 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 802136 T1810 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-005/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 802137 T1810 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=372707240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 802138 T1810 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@113c73f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@928f2c),segFN=segments_1,generation=1}
   [junit4]   2> 802138 T1810 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 802143 T1810 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 802143 T1810 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 802144 T1810 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 802144 T1810 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802145 T1810 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802145 T1810 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 802146 T1810 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802146 T1810 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802147 T1810 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 802148 T1810 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 802149 T1810 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 802149 T1810 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 802149 T1810 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 802150 T1810 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 802151 T1810 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 802152 T1810 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 802156 T1810 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 802168 T1810 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 802169 T1810 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 802169 T1810 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.904296875, noCFSRatio=0.5483011668453267]
   [junit4]   2> 802170 T1810 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@113c73f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@928f2c),segFN=segments_1,generation=1}
   [junit4]   2> 802170 T1810 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 802171 T1810 oass.SolrIndexSearcher.<init> Opening Searcher@9a3b33[solrj_collection_shard1_replica2] main
   [junit4]   2> 802171 T1810 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 802172 T1810 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 802172 T1810 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 802173 T1810 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 802173 T1810 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 802173 T1810 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 802174 T1810 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 802174 T1810 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 802175 T1810 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 802175 T1810 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 802178 T1838 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@9a3b33[solrj_collection_shard1_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 802180 T1810 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 802180 T1810 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:46138 collection:solrj_collection shard:shard1
   [junit4]   2> 802181 T1810 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 802189 T1810 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 802190 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 802191 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 802191 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 802191 T1810 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=7.6972726E14ms
   [junit4]   2> 802192 T1772 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 802194 T1771 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 802195 T1771 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802195 T1817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802195 T1789 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802195 T1776 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802195 T1831 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802195 T1803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 802231 T1766 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802231 T1766 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802232 T1766 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 802239 T1783 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802239 T1783 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 802240 T1783 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 802242 T1766 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 802246 T1783 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 802247 T1766 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 802249 T1783 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 802258 T1783 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802258 T1766 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802261 T1783 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802261 T1766 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 802265 T1783 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802265 T1766 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802266 T1783 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802266 T1783 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802266 T1766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802266 T1766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802266 T1783 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802267 T1766 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 802268 T1783 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802268 T1783 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802268 T1766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 802268 T1766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 802268 T1783 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 802269 T1766 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 802269 T1783 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 802269 T1766 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 802270 T1783 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/data/
   [junit4]   2> 802270 T1766 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/data/
   [junit4]   2> 802270 T1783 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a2e5c6
   [junit4]   2> 802271 T1766 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a2e5c6
   [junit4]   2> 802272 T1783 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/data
   [junit4]   2> 802272 T1766 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/data
   [junit4]   2> 802272 T1783 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 802272 T1766 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 802273 T1783 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 802273 T1766 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 802274 T1783 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-003/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 802274 T1766 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-5094A28D27BA6506-001/tempDir-002/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 802275 T1783 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=372707240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 802275 T1766 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=372707240, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 802282 T1783 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@18148c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@47e99a),segFN=segments_1,generation=1}
   [junit4]   2> 802283 T1766 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@101cf02 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1dfde71),segFN=segments_1,generation=1}
   [junit4]   2> 802283 T1766 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 802283 T1783 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 802286 T1766 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 802286 T1783 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 802287 T1766 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 802287 T1783 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 802287 T1766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 802288 T1783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 802288 T1766 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802288 T1783 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802289 T1766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802289 T1783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 802289 T1766 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 802289 T1783 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 802290 T1766 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802290 T1783 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802291 T1766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802291 T1766 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 802291 T1783 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 802292 T1783 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 802292 T1766 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 802292 T1783 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 802293 T1766 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 802293 T1766 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 802293 T1783 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 802294 T1783 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 802293 T1766 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 802294 T1783 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 802295 T1766 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 802295 T1783 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 802295 T1766 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 802295 T1783 oasc.RequestHandlers.init

[...truncated too long message...]

  2> 
   [junit4]   2> 1079919 T1904 oaz.ClientCnxn$SendThread.run WARN Session 0x14624c2947f000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 1080419 T1904 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]   2> 
   [junit4]   2> 1080919 T1904 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 1080920 T1904 oaz.ClientCnxn$SendThread.run WARN Session 0x14624c2947f000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 1081419 T1904 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]   2> 
   [junit4]   2> 1081919 T1904 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 1081920 T1904 oaz.ClientCnxn$SendThread.run WARN Session 0x14624c2947f000e for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> 289980 T1748 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=fr_FR, timezone=Australia/Currie
   [junit4]   2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.7.0_55 (32-bit)/cpus=8,threads=2,free=80919280,total=197836800
   [junit4]   2> NOTE: All tests run in this JVM: [MultiThreadedOCPTest, DocValuesMultiTest, TestFastLRUCache, TestSweetSpotSimilarityFactory, TestStressRecovery, DefaultValueUpdateProcessorTest, TestRandomMergePolicy, DeleteShardTest, TestCollationKeyRangeQueries, ShardSplitTest, XmlUpdateRequestHandlerTest, HdfsLockFactoryTest, TestDFRSimilarityFactory, RequiredFieldsTest, JSONWriterTest, TestHashPartitioner, TestQuerySenderListener, TestSchemaResource, OpenCloseCoreStressTest, ZkNodePropsTest, TestCodecSupport, TestSchemaVersionResource, StressHdfsTest, TestSolrCoreProperties, StatsComponentTest, CollectionsAPIAsyncDistributedZkTest, TestDistributedSearch, TestExpandComponent, TestBinaryResponseWriter, BasicDistributedZk2Test, TestExceedMaxTermLength, HdfsSyncSliceTest, AliasIntegrationTest, HighlighterTest, NotRequiredUniqueKeyTest, TestNonDefinedSimilarityFactory, TestSolrXml, MultiTermTest, TestCollectionAPI, TestStressReorder, TestRangeQuery, TestUtils, ConvertedLegacyTest, TestPartialUpdateDeduplication, CollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.seed=5094A28D27BA6506 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_FR -Dtests.timezone=Australia/Currie -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | CollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=17 closes=16
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:436)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:182)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   [junit4]    >    1) Thread[id=1905, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    2) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=1904, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[5094A28D27BA6506]-SendThread(127.0.0.1:56904), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5094A28D27BA6506]:0)
   [junit4] Completed on J1 in 290.02s, 1 test, 1 failure, 3 errors <<< FAILURES!

[...truncated 747 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 3 suite-level errors, 1 error, 954 ignored (12 assumptions)

Total time: 60 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_55 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b15) - Build # 10241 - Still Failing!

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
I have seen a similar error in another jenkins test run. I think it may
have something to do with SOLR-5681.


On Fri, May 23, 2014 at 2:20 AM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10241/
> Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC
>
> 1 tests failed.
> FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch
>
> Error Message:
> KeeperErrorCode = NodeExists for
> /overseer/collection-queue-work/qnr-0000000002
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> KeeperErrorCode = NodeExists for
> /overseer/collection-queue-work/qnr-0000000002
>         at
> __randomizedtesting.SeedInfo.seed([45878A30BD887025:C4610428CAD71019]:0)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
>         at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1508)
>         at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1468)
>         at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1527)
>         at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1464)
>         at
> org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
>         at
> org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
>         at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 11701 lines...]
>    [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
>    [junit4]   2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/init-core-data-001
>    [junit4]   2> 1556649 T6185
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /
>    [junit4]   2> 1556652 T6185 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
>    [junit4]   2> 1556652 T6185 oasc.ZkTestServer.run STARTING ZK TEST
> SERVER
>    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 1556653 T6186
> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>    [junit4]   2> 1556753 T6185 oasc.ZkTestServer.run start zk server on
> port:45460
>    [junit4]   2> 1556754 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1574469 T6189 oazsp.FileTxnLog.commit WARN fsync-ing the
> write ahead log in SyncThread:0 took 17714ms which will adversely effect
> operation latency. See the ZooKeeper troubleshooting guide
>    [junit4]   2> 1574471 T6187 oazs.NIOServerCnxn.doIO WARN caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x14625af03a50000, likely client has closed socket
>    [junit4]   2>        at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>    [junit4]   2>        at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>    [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2>
>    [junit4]   2> 1574474 T6192 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@217e711cname:ZooKeeperConnection Watcher:
> 127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1574474 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1574475 T6185 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 1574512 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1574532 T6194 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@34c581b2name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1574532 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1574532 T6185 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>    [junit4]   2> 1574567 T6185 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
>    [junit4]   2> 1574592 T6185 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
>    [junit4]   2> 1574612 T6185 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
>    [junit4]   2> 1574632 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
>    [junit4]   2> 1574633 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
>    [junit4]   2> 1576324 T6189 oazsp.FileTxnLog.commit WARN fsync-ing the
> write ahead log in SyncThread:0 took 1650ms which will adversely effect
> operation latency. See the ZooKeeper troubleshooting guide
>    [junit4]   2> 1576325 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml
> to /configs/conf1/schema.xml
>    [junit4]   2> 1576326 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
>    [junit4]   2> 1576328 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 1576329 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 1576331 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
>    [junit4]   2> 1576331 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
>    [junit4]   2> 1576333 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
>    [junit4]   2> 1576333 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
>    [junit4]   2> 1576335 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
>    [junit4]   2> 1576335 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
>    [junit4]   2> 1576337 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
>    [junit4]   2> 1576337 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
>    [junit4]   2> 1576340 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 1576340 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 1576343 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 1576343 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 1576345 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
>    [junit4]   2> 1576346 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
>    [junit4]   2> 1576347 T6185 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
>    [junit4]   2> 1576347 T6185 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
>    [junit4]   2> 1576628 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 1576631 T6185 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:42446
>    [junit4]   2> 1576631 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002
>    [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/'
>    [junit4]   2> 1576645 T6185 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/solr.xml
>    [junit4]   2> 1576701 T6185 oasc.CoreContainer.<init> New CoreContainer
> 105707606
>    [junit4]   2> 1576701 T6185 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/]
>    [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 1576704 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 1576704 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 1576704 T6185 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 1576705 T6185 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 1576705 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 1576705 T6185 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:45460/solr
>    [junit4]   2> 1576706 T6185 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 1576706 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1576708 T6205 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2743d955name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1576709 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1576710 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1576712 T6207 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@69a677cename:ZooKeeperConnection Watcher:
> 127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1576712 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1576713 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>    [junit4]   2> 1576715 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 1576717 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
>    [junit4]   2> 1576719 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 1576735 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 1576737 T6185 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>    [junit4]   2> 1576738 T6185 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45299_
>    [junit4]   2> 1576739 T6185 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:45299_
>    [junit4]   2> 1576741 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>    [junit4]   2> 1576742 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>    [junit4]   2> 1576744 T6185
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:45299_
>    [junit4]   2> 1576745 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>    [junit4]   2> 1576746 T6185 oasc.Overseer.start Overseer
> (id=91802276018323460-127.0.0.1:45299_-n_0000000000) starting
>    [junit4]   2> 1576748 T6185 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>    [junit4]   2> 1576757 T6209 oasc.OverseerCollectionProcessor.run
> Process current queue of collection creations
>    [junit4]   2> 1576757 T6185 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>    [junit4]   2> 1576759 T6185 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>    [junit4]   2> 1576760 T6185
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 1576762 T6208 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
>    [junit4]   2> 1576765 T6210 oasc.ZkController.publish publishing
> core=collection1 state=down collection=control_collection
>    [junit4]   2> 1576765 T6210 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1576766 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1576766 T6210 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 1576766 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1576767 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "base_url":"http://127.0.0.1:45299",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:45299_",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "numShards":"1",
>    [junit4]   2>          "core_node_name":null}
>    [junit4]   2> 1576767 T6208
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> control_collection with shards [shard1]
>    [junit4]   2> 1576767 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 1576769 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1576770 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 1577766 T6210 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 1577766 T6210 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:control_collection
>    [junit4]   2> 1577767 T6210 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 1577767 T6210 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 1577768 T6210 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 1577768 T6210 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/'
>    [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/README'
> to classloader
>    [junit4]   2> 1577805 T6210 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_4_9
>    [junit4]   2> 1577836 T6210 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 1577837 T6210 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 1577844 T6210 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 1578144 T6210 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1578144 T6210 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1578145 T6210 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 1578150 T6210 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 1578152 T6210 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 1578162 T6210 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1578165 T6210 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1578168 T6210 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1578169 T6210
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1578170 T6210
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1578170 T6210 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1578170 T6210
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1578171 T6210
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1578171 T6210 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection control_collection
>    [junit4]   2> 1578171 T6210 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 1578171 T6210 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/,
> dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/
>    [junit4]   2> 1578172 T6210 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
>    [junit4]   2> 1578172 T6210 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data
>    [junit4]   2> 1578172 T6210 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index/
>    [junit4]   2> 1578173 T6210 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 1578173 T6210 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index
>    [junit4]   2> 1578173 T6210 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42,
> maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875,
> forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
>    [junit4]   2> 1578174 T6210 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@6a607708lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62d7449e
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1578174 T6210 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1578179 T6210 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 1578181 T6210 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 1578183 T6210 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 1578185 T6210 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 1578185 T6210 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 1578186 T6210 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 1578186 T6210 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@6a607708lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62d7449e
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1578186 T6210 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1578187 T6210 oass.SolrIndexSearcher.<init> Opening
> Searcher@4502768b[collection1] main
>    [junit4]   2> 1578187 T6210 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 1578188 T6210 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 1578188 T6210 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 1578188 T6210
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 1578188 T6210 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 1578188 T6210 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1578189 T6210
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 1578189 T6210 oasr.ManagedResourceStorage.load Loaded
> null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1578189 T6210
> oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers
> for /rest/managed
>    [junit4]   2> 1578189 T6210 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 1578191 T6211 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@4502768b[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
>    [junit4]   2> 1578192 T6210 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 1578193 T6214 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:45299collection:control_collection shard:shard1
>    [junit4]   2> 1578193 T6185 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 1578194 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 1578196 T6214 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
>    [junit4]   2> 1578196 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1578199 T6217 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@669f2a28name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1578199 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1578200 T6185
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 1578202 T6214
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 1578202 T6185 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
>    [junit4]   2> 1578205 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578206 T6214
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 1578206 T6214
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> 1578206 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> ASYNC  NEW_CORE C1393 name=collection1
> org.apache.solr.core.SolrCore@5ab2bf70 url=
> http://127.0.0.1:45299/collection1 node=127.0.0.1:45299_
> C1393_STATE=coll:control_collection core:collection1 props:{state=down,
> base_url=http://127.0.0.1:45299, core=collection1,
> node_name=127.0.0.1:45299_}
>    [junit4]   2> 1578207 T6214 C1393 P45299 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:45299/collection1/
>    [junit4]   2> 1578208 T6214 C1393 P45299 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 1578208 T6214 C1393 P45299 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:45299/collection1/ has no replicas
>    [junit4]   2> 1578208 T6214
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:45299/collection1/ shard1
>    [junit4]   2> 1578208 T6214 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
>    [junit4]   2> 1578209 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578213 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578216 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578317 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 1578317 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 1578364 T6214 oasc.ZkController.register We are
> http://127.0.0.1:45299/collection1/ and leader is
> http://127.0.0.1:45299/collection1/
>    [junit4]   2> 1578365 T6214 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=http://127.0.0.1:45299
>    [junit4]   2> 1578365 T6214 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 1578365 T6214 oasc.ZkController.publish publishing
> core=collection1 state=active collection=control_collection
>    [junit4]   2> 1578365 T6214 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1578366 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578366 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578367 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578367 T6214 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1578368 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1578369 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "base_url":"http://127.0.0.1:45299",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:45299_",
>    [junit4]   2>          "shard":"shard1",
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "core_node_name":"core_node1"}
>    [junit4]   2> 1578370 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1578440 T6185
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>    [junit4]   2> 1578441 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 1578442 T6185 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:40851
>    [junit4]   2> 1578443 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 1578443 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 1578443 T6185 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003
>    [junit4]   2> 1578444 T6185 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/'
>    [junit4]   2> 1578456 T6185 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/solr.xml
>    [junit4]   2> 1578472 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 1578472 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 1578482 T6185 oasc.CoreContainer.<init> New CoreContainer
> 1628293887
>    [junit4]   2> 1578482 T6185 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/]
>    [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 1578485 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 1578485 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 1578485 T6185 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 1578486 T6185 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 1578486 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 1578486 T6185 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:45460/solr
>    [junit4]   2> 1578486 T6185 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 1578487 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1578488 T6228 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@508a5d20name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1578489 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1578490 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1578491 T6230 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@adb9a26name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1578491 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1578495 T6185
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 1579497 T6185 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44949_
>    [junit4]   2> 1579509 T6185 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:44949_
>    [junit4]   2> 1579512 T6217 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>    [junit4]   2> 1579512 T6207 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>    [junit4]   2> 1579512 T6230 oascc.ZkStateReader$3.process Updating live
> nodes... (2)
>    [junit4]   2> 1579517 T6231 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 1579517 T6231 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1579518 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1579518 T6231 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 1579518 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1579518 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1579519 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1579519 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "base_url":"http://127.0.0.1:44949",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:44949_",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "core_node_name":null}
>    [junit4]   2> 1579519 T6208
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2]
>    [junit4]   2> 1579519 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 1579521 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1579622 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1579622 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1579622 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1580518 T6231 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 1580519 T6231 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 1580519 T6231 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 1580519 T6231 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 1580520 T6231 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 1580521 T6231 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/'
>    [junit4]   2> 1580521 T6231 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 1580522 T6231 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 1580522 T6231 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/README'
> to classloader
>    [junit4]   2> 1580560 T6231 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_4_9
>    [junit4]   2> 1580588 T6231 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 1580589 T6231 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 1580596 T6231 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 1580852 T6231 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1580853 T6231 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1580853 T6231 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 1580858 T6231 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 1580860 T6231 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 1580908 T6231 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1580911 T6231 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1580913 T6231 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1580915 T6231
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1580915 T6231
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1580915 T6231 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1580916 T6231
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1580916 T6231
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1580916 T6231 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 1580916 T6231 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 1580917 T6231 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/,
> dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/
>    [junit4]   2> 1580917 T6231 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
>    [junit4]   2> 1580917 T6231 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1
>    [junit4]   2> 1580917 T6231 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index/
>    [junit4]   2> 1580918 T6231 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 1580918 T6231 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index
>    [junit4]   2> 1580918 T6231 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42,
> maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875,
> forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
>    [junit4]   2> 1580919 T6231 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@4215316clockFactory=org.apache.lucene.store.SingleInstanceLockFactory@474e2d26
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1580919 T6231 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1580924 T6231 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1580924 T6231 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 1580924 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 1580926 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 1580926 T6231 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 1580928 T6231 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 1580930 T6231 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 1580931 T6231 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 1580931 T6231 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 1580932 T6231 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@4215316clockFactory=org.apache.lucene.store.SingleInstanceLockFactory@474e2d26
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1580932 T6231 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1580932 T6231 oass.SolrIndexSearcher.<init> Opening
> Searcher@79c2bae[collection1] main
>    [junit4]   2> 1580933 T6231 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 1580933 T6231 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 1580934 T6231 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 1580934 T6231
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 1580934 T6231 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 1580934 T6231 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1580935 T6231
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 1580935 T6231 oasr.ManagedResourceStorage.load Loaded
> null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1580935 T6231
> oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers
> for /rest/managed
>    [junit4]   2> 1580935 T6231 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 1580938 T6232 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@79c2bae[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
>    [junit4]   2> 1580940 T6231 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 1580940 T6235 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:44949 collection:collection1
> shard:shard1
>    [junit4]   2> 1580941 T6185 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 1580941 T6235 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
>    [junit4]   2> 1580941 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 1580945 T6235
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 1580946 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1580946 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1580946 T6235
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 1580947 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1580947 T6235
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C1394 name=collection1
> org.apache.solr.core.SolrCore@4dd4cf8e url=
> http://127.0.0.1:44949/collection1 node=127.0.0.1:44949_
> C1394_STATE=coll:collection1 core:collection1 props:{state=down, base_url=
> http://127.0.0.1:44949, core=collection1, node_name=127.0.0.1:44949_}
>    [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:44949/collection1/
>    [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:44949/collection1/ has no replicas
>    [junit4]   2> 1580948 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1580948 T6235
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:44949/collection1/ shard1
>    [junit4]   2> 1580948 T6235 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
>    [junit4]   2> 1580952 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1580954 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1580958 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1581060 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581060 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581060 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581105 T6235 oasc.ZkController.register We are
> http://127.0.0.1:44949/collection1/ and leader is
> http://127.0.0.1:44949/collection1/
>    [junit4]   2> 1581106 T6235 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=http://127.0.0.1:44949
>    [junit4]   2> 1581106 T6235 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 1581106 T6235 oasc.ZkController.publish publishing
> core=collection1 state=active collection=collection1
>    [junit4]   2> 1581106 T6235 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1581107 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1581108 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1581108 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1581108 T6235 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1581109 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1581109 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "base_url":"http://127.0.0.1:44949",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:44949_",
>    [junit4]   2>          "shard":"shard1",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "core_node_name":"core_node1"}
>    [junit4]   2> 1581111 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1581196 T6185
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>    [junit4]   2> 1581197 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 1581198 T6185 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:58625
>    [junit4]   2> 1581199 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 1581199 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 1581200 T6185 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004
>    [junit4]   2> 1581200 T6185 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/'
>    [junit4]   2> 1581212 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581212 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581212 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 1581216 T6185 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/solr.xml
>    [junit4]   2> 1581243 T6185 oasc.CoreContainer.<init> New CoreContainer
> 1092161998
>    [junit4]   2> 1581243 T6185 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/]
>    [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 1581246 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 1581246 T6185 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 1581246 T6185 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 1581247 T6185 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 1581247 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 1581248 T6185 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:45460/solr
>    [junit4]   2> 1581248 T6185 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 1581249 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1581250 T6247 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@54e00cc5name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1581251 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1581252 T6185 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 1581253 T6249 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6a970081name:ZooKeeperConnection Watcher:
> 127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None
> path:null path:null type:None
>    [junit4]   2> 1581253 T6185 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 1581258 T6185
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 1582260 T6185 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48572_
>    [junit4]   2> 1582268 T6185 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:48572_
>    [junit4]   2> 1582270 T6230 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>    [junit4]   2> 1582270 T6207 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>    [junit4]   2> 1582271 T6217 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>    [junit4]   2> 1582270 T6249 oascc.ZkStateReader$3.process Updating live
> nodes... (3)
>    [junit4]   2> 1582276 T6250 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 1582276 T6250 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1582277 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1582277 T6250 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 1582277 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1582277 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1582278 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1582278 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "base_url":"http://127.0.0.1:48572",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:48572_",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "core_node_name":null}
>    [junit4]   2> 1582279 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=2
>    [junit4]   2> 1582279 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
>    [junit4]   2> 1582281 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1582382 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1582382 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1582382 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1582382 T6249 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583277 T6250 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 1583277 T6250 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 1583278 T6250 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 1583278 T6250 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 1583279 T6250 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 1583279 T6250 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/'
>    [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/README'
> to classloader
>    [junit4]   2> 1583315 T6250 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: LUCENE_4_9
>    [junit4]   2> 1583339 T6250 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 1583340 T6250 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 1583347 T6250 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 1583610 T6250 oass.ByteField.init WARN ByteField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1583610 T6250 oass.ShortField.init WARN ShortField is
> deprecated and will be removed in 5.0. You should use TrieIntField instead.
>    [junit4]   2> 1583611 T6250 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 1583616 T6250 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 1583618 T6250 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 1583627 T6250 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1583629 T6250 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 1583632 T6250 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1583633 T6250
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1583633 T6250
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1583634 T6250 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 1583634 T6250
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 1583634 T6250
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 1583635 T6250 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 1583635 T6250 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 1583635 T6250 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/,
> dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/
>    [junit4]   2> 1583635 T6250 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
>    [junit4]   2> 1583636 T6250 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2
>    [junit4]   2> 1583636 T6250 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index/
>    [junit4]   2> 1583636 T6250 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 1583636 T6250 oasc.CachingDirectoryFactory.get return new
> directory for
> ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index
>    [junit4]   2> 1583637 T6250 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42,
> maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875,
> forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
>    [junit4]   2> 1583637 T6250 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@4e58aa1elockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12194794
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1583638 T6250 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1583640 T6250 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 1583642 T6250 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 1583643 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 1583643 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 1583645 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 1583645 T6250 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 1583648 T6250 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 1583650 T6250 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 1583651 T6250 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 1583651 T6250 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 1583652 T6250 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
>  commit{dir=MockDirectoryWrapper(RAMDirectory@4e58aa1elockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12194794
> ),segFN=segments_1,generation=1}
>    [junit4]   2> 1583652 T6250 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 1583652 T6250 oass.SolrIndexSearcher.<init> Opening
> Searcher@14adffcc[collection1] main
>    [junit4]   2> 1583652 T6250 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 1583653 T6250 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 1583653 T6250 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 1583654 T6250
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 1583654 T6250 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 1583654 T6250 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1583654 T6250
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 1583655 T6250 oasr.ManagedResourceStorage.load Loaded
> null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 1583655 T6250
> oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers
> for /rest/managed
>    [junit4]   2> 1583655 T6250 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 1583658 T6251 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@14adffcc[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
>    [junit4]   2> 1583659 T6250 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 1583660 T6254 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:48572 collection:collection1
> shard:shard2
>    [junit4]   2> 1583660 T6185 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
>    [junit4]   2> 1583661 T6185 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 1583662 T6254 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
>    [junit4]   2> 1583662 T6185 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1583663 T6185
> oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for
> recoveries to finish - wait 30000 for each attempt
>    [junit4]   2> 1583663 T6185
> oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for
> recoveries to finish - collection: collection1 failOnTimeout:true timeout
> (sec):30000
>    [junit4]   2> 1583663 T6185 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1583666 T6254
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard2
>    [junit4]   2> 1583668 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583669 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583669 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583669 T6254
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 1583669 T6254
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C1395 name=collection1
> org.apache.solr.core.SolrCore@60d7ab8b url=
> http://127.0.0.1:48572/collection1 node=127.0.0.1:48572_
> C1395_STATE=coll:collection1 core:collection1 props:{state=down, base_url=
> http://127.0.0.1:48572, core=collection1, node_name=127.0.0.1:48572_}
>    [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:48572/collection1/
>    [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:48572/collection1/ has no replicas
>    [junit4]   2> 1583670 T6254
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:48572/collection1/ shard2
>    [junit4]   2> 1583670 T6254 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
>    [junit4]   2> 1583671 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1583673 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583675 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583678 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583779 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583779 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583779 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583779 T6249 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583826 T6254 oasc.ZkController.register We are
> http://127.0.0.1:48572/collection1/ and leader is
> http://127.0.0.1:48572/collection1/
>    [junit4]   2> 1583826 T6254 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=http://127.0.0.1:48572
>    [junit4]   2> 1583826 T6254 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 1583827 T6254 oasc.ZkController.publish publishing
> core=collection1 state=active collection=collection1
>    [junit4]   2> 1583827 T6254 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 1583830 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583830 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583830 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583831 T6254 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1583832 T6208 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1583832 T6208
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "base_url":"http://127.0.0.1:48572",
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "node_name":"127.0.0.1:48572_",
>    [junit4]   2>          "shard":"shard2",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "core_node_name":"core_node2"}
>    [junit4]   2> 1583834 T6207
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 1583937 T6230 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583937 T6217 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583937 T6249 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1583937 T6207 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 1584665 T6185 oascc.ZkStateReader.updateClusterState
> Updating cloud state from ZooKeeper...
>    [junit4]   2> 1584666 T6185
> oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries
> finished - collection: collection1
>    [junit4]   2> 1584667 T6215 oasc.SocketProxy$Acceptor.run accepted
> Socket[addr=/127.0.0.1,port=56586,localport=45299],
> receiveBufferSize:131015
>    [junit4]   2> 1584669 T6215 oasc.SocketProxy$Bridge.<init> proxy
> connection Socket[addr=/127.0.0.1,port=42446,localport=43298],
> receiveBufferSize=131003
>    [junit4]   2> ASYNC  NEW_CORE C1396 name=collection1
> org.apache.solr.core.SolrCore@5ab2bf70 url=
> http://127.0.0.1:45299/collection1 node=127.0.0.1:45299_
> C1396_STATE=coll:control_collection core:collection1 props:{state=active,
> base_url=http://127.0.0.1:45299, core=collection1,
> node_name=127.0.0.1:45299_, leader=true}
>    [junit4]   2> 1584671 T6201 C1396 P45299
> oasu.DirectUpdateHandler2.commit start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 1584672 T6201 C1396 P45299
> oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
>    [junit4]   2> 1584672 T6201 C1396 P45299 oasc.SolrCore.openNewSearcher
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
>    [junit4]   2> 1584672 T6201 C1396 P45299
> oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 1584673 T6201 C1396 P45299
> oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update
> params={commit=true&version=2&wt=javabin&waitSearcher=true&softCommit=false}
> {commit=} 0 2
>    [junit4]   2> 1584674 T6236 oasc.SocketProxy$Acceptor.run accepted
> Socket[addr=/127.0.0.1,port=46220,localport=44949],
> receiveBufferSize:131015
>    [junit4]   2> 1584675 T6236 oasc.SocketProxy$Bridge.<init> proxy
> connection Socket[addr=/127.0.0.1,port=40851,localport=54498],
> receiveBufferSize=131003
>    [junit4]   2> 1584677 T6236 oasc.SocketProxy$Acceptor.run accepted
> Socket[addr=/127.0.0.1,port=46222,localport=44949],
> receiveBufferSize:131015
>    [junit4]   2> 1584678 T6236 oasc.SocketProxy$Bridge.<init> proxy
> connection Socket[addr=/127.0.0.1,port=40851,localport=54500],
> receiveBufferSize=131003
>    [junit4]   2> ASYNC  NEW_CORE C1397 name=collection1
> org.apache.solr.core.SolrCore@4dd4cf8e url=
> http://127.0.0.1:44949/collection1 node=127.0.0.1:44949_
> C1397_STATE=coll:collection1 core:collection1 props:{state=active, base_url=
> http://127.0.0.1:44949, core=collection1, node_name=127.0.0.1:44949_,
> leader=true}
>    [junit4]   2> 1584679 T6223 C1397 P44949
> oasu.DirectUpdateHandler2.commit start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 1584679 T6223 C1397 P44949
> oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
>    [junit4]   2> 1584679 T6223 C1397 P44949 oasc.SolrCore.openNewSearcher
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
>    [junit4]   2> 1584679 T6223 C1397 P44949
> oasu.DirectUpdateHandler2.commit end_commit_flush
>    [junit4]   2> 1584680 T6223 C1397 P44949
> oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update
> params={commit=true&commit_end_point=true&expungeDeletes=false&version=2&distrib.from=
> http://127.0.0.1:44949/collection1/&update.distrib=FROMLEADER&wt=javabin&waitSearcher=true&softCommit=false&openSearcher=true}
> {commit=} 0 2
>    [junit4]   2> 1584681 T6255 oasc.SocketProxy$Acceptor.run accepted
> Socket[addr=/127.0.0.1,port=44016,localport=48572],
> receiveBufferSize:131015
>    [junit4]   2> 1584681 T6255 oasc.SocketProxy$Bridge.<init> proxy
> connection Socket[addr=/127.0.0.1,port=58625,localport=41081],
> receiveBufferSize=131003
>    [junit4]   2>  C1395_STATE=coll:collection1 core:collection1
> props:{state=active, base_url=http://127.0.0.1:48572, core=collection1,
> node_name=127.0.0.1:48572_, leader=true}
>    [junit4]   2> 1584682 T6243 C1395 P48572
> oasu.DirectUpdateHandler2.commit start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>    [junit4]   2> 1584682 T6243 C1395 P48572
> oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
>    [junit4]   2> 1584682 T6243 C1395 P48572 oasc.SolrCore.openNewSearcher
> SolrIndexSearcher has not changed - not re-opening:
> org.apache.solr.search.SolrIndexSearcher
>    [junit4]   2> 1584683 T6243
>
> [...truncated too long message...]
>
> .timezone=Asia/Aqtau -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR    118s J1 | HttpPartitionTest.testDistribSearch <<<
>    [junit4]    > Throwable #1:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> KeeperErrorCode = NodeExists for
> /overseer/collection-queue-work/qnr-0000000002
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([45878A30BD887025:C4610428CAD71019]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1508)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1468)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1527)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1464)
>    [junit4]    >        at
> org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
>    [junit4]    >        at
> org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
>    [junit4]    >        at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> 1675063 T6185 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001
>    [junit4]   2> 118420 T6184 ccr.ThreadLeakControl.checkThreadLeaks
> WARNING Will linger awaiting termination of 5 leaked thread(s).
>    [junit4]   2> 1675744 T6207 oascc.ZkStateReader$3.process WARN
> ZooKeeper watch triggered, but Solr cannot talk to ZK
>    [junit4]   2> 1675841 T6343
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
>    [junit4]   2>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>    [junit4]   2>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>    [junit4]   2>        at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
>    [junit4]   2>
>    [junit4]   2> 1675841 T6230 oascc.ZkStateReader$3.process WARN
> ZooKeeper watch triggered, but Solr cannot talk to ZK
>    [junit4]   2> 1675925 T6347
> oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not
> read the data org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer_elect/leader
>    [junit4]   2>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>    [junit4]   2>        at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>    [junit4]   2>        at
> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>    [junit4]   2>        at
> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
>    [junit4]   2>        at
> org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
>    [junit4]   2>
>    [junit4]   2> NOTE: test params are: codec=Lucene40,
> sim=DefaultSimilarity, locale=ar_BH, timezone=Asia/Aqtau
>    [junit4]   2> NOTE: Linux 3.8.0-39-generic amd64/Oracle Corporation
> 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=48390064,total=256901120
>    [junit4]   2> NOTE: All tests run in this JVM: [TestSolrXmlPersistence,
> TestNRTOpen, DirectSolrConnectionTest, TestWordDelimiterFilterFactory,
> TestRangeQuery, CachingDirectoryFactoryTest, PluginInfoTest,
> SuggesterTSTTest, HdfsWriteToMultipleCollectionsTest,
> StandardRequestHandlerTest, TestSolrXml, TestAtomicUpdateErrorCases,
> TestSimpleQParserPlugin, TestManagedSynonymFilterFactory, StressHdfsTest,
> BasicDistributedZk2Test, TestPhraseSuggestions, TestBinaryResponseWriter,
> TestUpdate, TestSchemaVersionResource, TestCloudManagedSchema,
> NotRequiredUniqueKeyTest, TestRecoveryHdfs, TestFoldingMultitermQuery,
> SolrIndexConfigTest, TestZkChroot, DefaultValueUpdateProcessorTest,
> TestShortCircuitedRequests, TestNumberUtils, ZkSolrClientTest,
> TestIndexSearcher, TestArbitraryIndexDir,
> SignatureUpdateProcessorFactoryTest, XsltUpdateRequestHandlerTest,
> TestRandomMergePolicy, TestAnalyzeInfixSuggestions, TestQueryTypes,
> TestFuzzyAnalyzedSuggestions, TestMiniSolrCloudCluster,
> IndexBasedSpellCheckerTest, ConnectionManagerTest, TestJmxIntegration,
> TestCSVResponseWriter, TestCustomSort, TestHashPartitioner,
> TestDefaultSimilarityFactory, TestQuerySenderNoQuery,
> TestSweetSpotSimilarityFactory, AliasIntegrationTest,
> TestExceedMaxTermLength, TestBinaryField, TestManagedSchema,
> TestFastLRUCache, TestShardHandlerFactory, ZkNodePropsTest,
> TestFieldTypeResource, TestSolrQueryParserResource, StatsComponentTest,
> TestRestManager, RequestHandlersTest, TestUtils, ShardSplitTest,
> TestCloudManagedSchemaAddField, TestSolrCoreProperties, SimplePostToolTest,
> RAMDirectoryFactoryTest, TestDistributedSearch, TestMultiCoreConfBootstrap,
> CursorPagingTest, DebugComponentTest, PreAnalyzedFieldTest,
> TestRandomFaceting, SortByFunctionTest, TestCollationField,
> DocValuesMultiTest, CollectionsAPIAsyncDistributedZkTest,
> TestCollationKeyRangeQueries, TermVectorComponentDistributedTest,
> CurrencyFieldOpenExchangeTest, TestRTGBase, TestCSVLoader,
> TestXIncludeConfig, TestDynamicFieldCollectionResource, HdfsSyncSliceTest,
> IndexSchemaRuntimeFieldTest, SpellCheckCollatorTest, MultiTermTest,
> TestInitQParser, TestFastOutputStream, TriLevelCompositeIdRoutingTest,
> TestConfigSets, DirectUpdateHandlerTest, HighlighterConfigTest,
> CollectionsAPIDistributedZkTest, NumericFieldsTest, TestFastWriter,
> LeaderElectionIntegrationTest, RecoveryZkTest, TestReplicationHandler,
> ShardRoutingCustomTest, TestDistributedGrouping, ZkControllerTest,
> TestStressReorder, TestReload, TestStressVersions, TestSolr4Spatial,
> SolrCmdDistributorTest, QueryElevationComponentTest, TestSort,
> TestFunctionQuery, TestLazyCores, TestBadConfig,
> OverseerCollectionProcessorTest, HighlighterTest, CurrencyFieldXmlFileTest,
> TestCoreDiscovery, AnalysisAfterCoreReloadTest, SolrRequestParserTest,
> SpatialFilterTest, SuggesterWFSTTest, NoCacheHeaderTest,
> SolrCoreCheckLockOnStartupTest, StatelessScriptUpdateProcessorFactoryTest,
> DistanceFunctionTest, CacheHeaderTest, DisMaxRequestHandlerTest,
> DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest,
> TestValueSourceCache, TermVectorComponentTest, TestIndexingPerformance,
> FieldAnalysisRequestHandlerTest, LoggingHandlerTest, SolrPluginUtilsTest,
> ReturnFieldsTest, MBeansHandlerTest, TestAnalyzedSuggestions,
> BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication,
> PingRequestHandlerTest, SearchHandlerTest, AlternateDirectoryTest,
> BadComponentTest, TestStressRecovery, SpellPossibilityIteratorTest,
> TestCodecSupport, EchoParamsTest, TestPerFieldSimilarity,
> TestIBSimilarityFactory, TestJmxMonitoredMap, ResourceLoaderTest,
> ChaosMonkeyNothingIsSafeTest, DeleteReplicaTest, DistribCursorPagingTest,
> HttpPartitionTest]
>    [junit4] Completed on J1 in 119.45s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 192 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921:
> There were test failures: 396 suites, 1640 tests, 1 error, 36 ignored (15
> assumptions)
>
> Total time: 66 minutes 40 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops
> -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
Regards,
Shalin Shekhar Mangar.

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b15) - Build # 10241 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10241/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
KeeperErrorCode = NodeExists for /overseer/collection-queue-work/qnr-0000000002

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: KeeperErrorCode = NodeExists for /overseer/collection-queue-work/qnr-0000000002
	at __randomizedtesting.SeedInfo.seed([45878A30BD887025:C4610428CAD71019]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1508)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1468)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1527)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1464)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11701 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/init-core-data-001
   [junit4]   2> 1556649 T6185 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1556652 T6185 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1556652 T6185 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1556653 T6186 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1556753 T6185 oasc.ZkTestServer.run start zk server on port:45460
   [junit4]   2> 1556754 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1574469 T6189 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 17714ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 1574471 T6187 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14625af03a50000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1574474 T6192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@217e711c name:ZooKeeperConnection Watcher:127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1574474 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1574475 T6185 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1574512 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1574532 T6194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34c581b2 name:ZooKeeperConnection Watcher:127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1574532 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1574532 T6185 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1574567 T6185 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1574592 T6185 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1574612 T6185 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1574632 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1574633 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1576324 T6189 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1650ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 1576325 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1576326 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1576328 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1576329 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1576331 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1576331 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1576333 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1576333 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1576335 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1576335 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1576337 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1576337 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1576340 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1576340 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1576343 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1576343 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1576345 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1576346 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1576347 T6185 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1576347 T6185 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1576628 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1576631 T6185 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42446
   [junit4]   2> 1576631 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002
   [junit4]   2> 1576632 T6185 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/'
   [junit4]   2> 1576645 T6185 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/solr.xml
   [junit4]   2> 1576701 T6185 oasc.CoreContainer.<init> New CoreContainer 105707606
   [junit4]   2> 1576701 T6185 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/]
   [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1576702 T6185 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1576703 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1576704 T6185 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1576704 T6185 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1576704 T6185 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1576705 T6185 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1576705 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1576705 T6185 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45460/solr
   [junit4]   2> 1576706 T6185 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1576706 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1576708 T6205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2743d955 name:ZooKeeperConnection Watcher:127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1576709 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1576710 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1576712 T6207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69a677ce name:ZooKeeperConnection Watcher:127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1576712 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1576713 T6185 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1576715 T6185 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1576717 T6185 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1576719 T6185 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1576735 T6185 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1576737 T6185 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1576738 T6185 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45299_
   [junit4]   2> 1576739 T6185 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45299_
   [junit4]   2> 1576741 T6185 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1576742 T6185 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1576744 T6185 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:45299_
   [junit4]   2> 1576745 T6185 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1576746 T6185 oasc.Overseer.start Overseer (id=91802276018323460-127.0.0.1:45299_-n_0000000000) starting
   [junit4]   2> 1576748 T6185 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1576757 T6209 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1576757 T6185 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1576759 T6185 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1576760 T6185 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1576762 T6208 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1576765 T6210 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1576765 T6210 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1576766 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1576766 T6210 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1576766 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1576767 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45299",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45299_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1576767 T6208 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1576767 T6208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1576769 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1576770 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1577766 T6210 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1577766 T6210 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1577767 T6210 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1577767 T6210 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1577768 T6210 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1577768 T6210 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/'
   [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1577769 T6210 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1577805 T6210 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1577836 T6210 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1577837 T6210 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1577844 T6210 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1578144 T6210 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1578144 T6210 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1578145 T6210 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1578150 T6210 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1578152 T6210 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1578162 T6210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1578165 T6210 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1578168 T6210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1578169 T6210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1578170 T6210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1578170 T6210 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1578170 T6210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1578171 T6210 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1578171 T6210 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1578171 T6210 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1578171 T6210 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/
   [junit4]   2> 1578172 T6210 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
   [junit4]   2> 1578172 T6210 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data
   [junit4]   2> 1578172 T6210 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index/
   [junit4]   2> 1578173 T6210 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1578173 T6210 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/control/data/index
   [junit4]   2> 1578173 T6210 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875, forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
   [junit4]   2> 1578174 T6210 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6a607708 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62d7449e),segFN=segments_1,generation=1}
   [junit4]   2> 1578174 T6210 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1578177 T6210 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1578178 T6210 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1578179 T6210 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1578179 T6210 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1578180 T6210 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1578181 T6210 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1578183 T6210 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1578185 T6210 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1578185 T6210 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1578186 T6210 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1578186 T6210 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6a607708 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62d7449e),segFN=segments_1,generation=1}
   [junit4]   2> 1578186 T6210 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1578187 T6210 oass.SolrIndexSearcher.<init> Opening Searcher@4502768b[collection1] main
   [junit4]   2> 1578187 T6210 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1578188 T6210 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1578188 T6210 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1578188 T6210 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1578188 T6210 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1578188 T6210 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1578189 T6210 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1578189 T6210 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1578189 T6210 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1578189 T6210 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1578191 T6211 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4502768b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1578192 T6210 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1578193 T6214 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45299 collection:control_collection shard:shard1
   [junit4]   2> 1578193 T6185 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1578194 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1578196 T6214 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1578196 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1578199 T6217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@669f2a28 name:ZooKeeperConnection Watcher:127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1578199 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1578200 T6185 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1578202 T6214 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1578202 T6185 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1578205 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578206 T6214 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1578206 T6214 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1578206 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1393 name=collection1 org.apache.solr.core.SolrCore@5ab2bf70 url=http://127.0.0.1:45299/collection1 node=127.0.0.1:45299_ C1393_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:45299, core=collection1, node_name=127.0.0.1:45299_}
   [junit4]   2> 1578207 T6214 C1393 P45299 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45299/collection1/
   [junit4]   2> 1578208 T6214 C1393 P45299 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1578208 T6214 C1393 P45299 oasc.SyncStrategy.syncToMe http://127.0.0.1:45299/collection1/ has no replicas
   [junit4]   2> 1578208 T6214 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45299/collection1/ shard1
   [junit4]   2> 1578208 T6214 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1578209 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578213 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578216 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578317 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1578317 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1578364 T6214 oasc.ZkController.register We are http://127.0.0.1:45299/collection1/ and leader is http://127.0.0.1:45299/collection1/
   [junit4]   2> 1578365 T6214 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45299
   [junit4]   2> 1578365 T6214 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1578365 T6214 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1578365 T6214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1578366 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578366 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578367 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578367 T6214 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1578368 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1578369 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45299",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45299_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1578370 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1578440 T6185 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1578441 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1578442 T6185 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40851
   [junit4]   2> 1578443 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1578443 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1578443 T6185 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003
   [junit4]   2> 1578444 T6185 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/'
   [junit4]   2> 1578456 T6185 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/solr.xml
   [junit4]   2> 1578472 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1578472 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1578482 T6185 oasc.CoreContainer.<init> New CoreContainer 1628293887
   [junit4]   2> 1578482 T6185 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/]
   [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1578483 T6185 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1578484 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1578485 T6185 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1578485 T6185 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1578485 T6185 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1578486 T6185 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1578486 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1578486 T6185 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45460/solr
   [junit4]   2> 1578486 T6185 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1578487 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1578488 T6228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@508a5d20 name:ZooKeeperConnection Watcher:127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1578489 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1578490 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1578491 T6230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@adb9a26 name:ZooKeeperConnection Watcher:127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1578491 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1578495 T6185 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1579497 T6185 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44949_
   [junit4]   2> 1579509 T6185 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44949_
   [junit4]   2> 1579512 T6217 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1579512 T6207 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1579512 T6230 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1579517 T6231 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1579517 T6231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1579518 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1579518 T6231 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1579518 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1579518 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1579519 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1579519 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44949",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44949_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1579519 T6208 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1579519 T6208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1579521 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1579622 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1579622 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1579622 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1580518 T6231 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1580519 T6231 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1580519 T6231 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1580519 T6231 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1580520 T6231 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1580521 T6231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/'
   [junit4]   2> 1580521 T6231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1580522 T6231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1580522 T6231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1580560 T6231 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1580588 T6231 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1580589 T6231 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1580596 T6231 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1580852 T6231 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1580853 T6231 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1580853 T6231 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1580858 T6231 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1580860 T6231 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1580908 T6231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1580911 T6231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1580913 T6231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1580915 T6231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1580915 T6231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1580915 T6231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1580916 T6231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1580916 T6231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1580916 T6231 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1580916 T6231 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1580917 T6231 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/
   [junit4]   2> 1580917 T6231 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
   [junit4]   2> 1580917 T6231 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1
   [junit4]   2> 1580917 T6231 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index/
   [junit4]   2> 1580918 T6231 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1580918 T6231 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty1/index
   [junit4]   2> 1580918 T6231 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875, forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
   [junit4]   2> 1580919 T6231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4215316c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@474e2d26),segFN=segments_1,generation=1}
   [junit4]   2> 1580919 T6231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1580922 T6231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1580923 T6231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1580924 T6231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1580924 T6231 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1580924 T6231 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1580925 T6231 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1580926 T6231 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1580926 T6231 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1580928 T6231 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1580930 T6231 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1580931 T6231 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1580931 T6231 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1580932 T6231 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4215316c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@474e2d26),segFN=segments_1,generation=1}
   [junit4]   2> 1580932 T6231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1580932 T6231 oass.SolrIndexSearcher.<init> Opening Searcher@79c2bae[collection1] main
   [junit4]   2> 1580933 T6231 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1580933 T6231 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1580934 T6231 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1580934 T6231 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1580934 T6231 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1580934 T6231 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1580935 T6231 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1580935 T6231 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1580935 T6231 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1580935 T6231 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1580938 T6232 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c2bae[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1580940 T6231 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1580940 T6235 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44949 collection:collection1 shard:shard1
   [junit4]   2> 1580941 T6185 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1580941 T6235 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1580941 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1580945 T6235 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1580946 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1580946 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1580946 T6235 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1580947 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1580947 T6235 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1394 name=collection1 org.apache.solr.core.SolrCore@4dd4cf8e url=http://127.0.0.1:44949/collection1 node=127.0.0.1:44949_ C1394_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:44949, core=collection1, node_name=127.0.0.1:44949_}
   [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44949/collection1/
   [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1580947 T6235 C1394 P44949 oasc.SyncStrategy.syncToMe http://127.0.0.1:44949/collection1/ has no replicas
   [junit4]   2> 1580948 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1580948 T6235 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44949/collection1/ shard1
   [junit4]   2> 1580948 T6235 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1580952 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1580954 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1580958 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1581060 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581060 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581060 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581105 T6235 oasc.ZkController.register We are http://127.0.0.1:44949/collection1/ and leader is http://127.0.0.1:44949/collection1/
   [junit4]   2> 1581106 T6235 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44949
   [junit4]   2> 1581106 T6235 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1581106 T6235 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1581106 T6235 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1581107 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1581108 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1581108 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1581108 T6235 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1581109 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1581109 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44949",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44949_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1581111 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1581196 T6185 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1581197 T6185 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1581198 T6185 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58625
   [junit4]   2> 1581199 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1581199 T6185 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1581200 T6185 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004
   [junit4]   2> 1581200 T6185 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/'
   [junit4]   2> 1581212 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581212 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581212 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1581216 T6185 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/solr.xml
   [junit4]   2> 1581243 T6185 oasc.CoreContainer.<init> New CoreContainer 1092161998
   [junit4]   2> 1581243 T6185 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/]
   [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1581244 T6185 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1581245 T6185 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1581246 T6185 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1581246 T6185 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1581246 T6185 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1581247 T6185 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1581247 T6185 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1581248 T6185 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45460/solr
   [junit4]   2> 1581248 T6185 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1581249 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1581250 T6247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54e00cc5 name:ZooKeeperConnection Watcher:127.0.0.1:45460 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1581251 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1581252 T6185 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1581253 T6249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a970081 name:ZooKeeperConnection Watcher:127.0.0.1:45460/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1581253 T6185 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1581258 T6185 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1582260 T6185 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48572_
   [junit4]   2> 1582268 T6185 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48572_
   [junit4]   2> 1582270 T6230 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1582270 T6207 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1582271 T6217 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1582270 T6249 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1582276 T6250 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1582276 T6250 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1582277 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1582277 T6250 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1582277 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1582277 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1582278 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1582278 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48572",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48572_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1582279 T6208 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1582279 T6208 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1582281 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1582382 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1582382 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1582382 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1582382 T6249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583277 T6250 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1583277 T6250 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1583278 T6250 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1583278 T6250 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1583279 T6250 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1583279 T6250 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/'
   [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1583280 T6250 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1583315 T6250 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1583339 T6250 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1583340 T6250 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1583347 T6250 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1583610 T6250 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1583610 T6250 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1583611 T6250 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1583616 T6250 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1583618 T6250 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1583627 T6250 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1583629 T6250 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1583632 T6250 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1583633 T6250 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1583633 T6250 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1583634 T6250 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1583634 T6250 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1583634 T6250 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1583635 T6250 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1583635 T6250 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1583635 T6250 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/
   [junit4]   2> 1583635 T6250 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30afd58d
   [junit4]   2> 1583636 T6250 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2
   [junit4]   2> 1583636 T6250 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index/
   [junit4]   2> 1583636 T6250 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1583636 T6250 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-45878A30BD887025-001/tempDir-001/jetty2/index
   [junit4]   2> 1583637 T6250 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=58.1484375, floorSegmentMB=2.1591796875, forceMergeDeletesPctAllowed=15.824503365522617, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18873968203764938
   [junit4]   2> 1583637 T6250 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e58aa1e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12194794),segFN=segments_1,generation=1}
   [junit4]   2> 1583638 T6250 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1583640 T6250 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1583641 T6250 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1583642 T6250 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1583642 T6250 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1583643 T6250 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1583643 T6250 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1583644 T6250 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1583645 T6250 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1583645 T6250 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1583648 T6250 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1583650 T6250 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1583651 T6250 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1583651 T6250 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1583652 T6250 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e58aa1e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12194794),segFN=segments_1,generation=1}
   [junit4]   2> 1583652 T6250 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1583652 T6250 oass.SolrIndexSearcher.<init> Opening Searcher@14adffcc[collection1] main
   [junit4]   2> 1583652 T6250 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1583653 T6250 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1583653 T6250 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1583654 T6250 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1583654 T6250 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1583654 T6250 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1583654 T6250 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1583655 T6250 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1583655 T6250 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1583655 T6250 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1583658 T6251 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14adffcc[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1583659 T6250 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1583660 T6254 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48572 collection:collection1 shard:shard2
   [junit4]   2> 1583660 T6185 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1583661 T6185 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1583662 T6254 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1583662 T6185 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1583663 T6185 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 1583663 T6185 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 1583663 T6185 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1583666 T6254 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1583668 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583669 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583669 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583669 T6254 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1583669 T6254 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1395 name=collection1 org.apache.solr.core.SolrCore@60d7ab8b url=http://127.0.0.1:48572/collection1 node=127.0.0.1:48572_ C1395_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:48572, core=collection1, node_name=127.0.0.1:48572_}
   [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48572/collection1/
   [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1583670 T6254 C1395 P48572 oasc.SyncStrategy.syncToMe http://127.0.0.1:48572/collection1/ has no replicas
   [junit4]   2> 1583670 T6254 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48572/collection1/ shard2
   [junit4]   2> 1583670 T6254 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1583671 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1583673 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583675 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583678 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583779 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583779 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583779 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583779 T6249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583826 T6254 oasc.ZkController.register We are http://127.0.0.1:48572/collection1/ and leader is http://127.0.0.1:48572/collection1/
   [junit4]   2> 1583826 T6254 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48572
   [junit4]   2> 1583826 T6254 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1583827 T6254 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1583827 T6254 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1583830 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583830 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583830 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583831 T6254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1583832 T6208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1583832 T6208 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48572",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48572_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1583834 T6207 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1583937 T6230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583937 T6217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583937 T6249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1583937 T6207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1584665 T6185 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1584666 T6185 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
   [junit4]   2> 1584667 T6215 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/127.0.0.1,port=56586,localport=45299], receiveBufferSize:131015
   [junit4]   2> 1584669 T6215 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=42446,localport=43298], receiveBufferSize=131003
   [junit4]   2> ASYNC  NEW_CORE C1396 name=collection1 org.apache.solr.core.SolrCore@5ab2bf70 url=http://127.0.0.1:45299/collection1 node=127.0.0.1:45299_ C1396_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:45299, core=collection1, node_name=127.0.0.1:45299_, leader=true}
   [junit4]   2> 1584671 T6201 C1396 P45299 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1584672 T6201 C1396 P45299 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1584672 T6201 C1396 P45299 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1584672 T6201 C1396 P45299 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1584673 T6201 C1396 P45299 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={commit=true&version=2&wt=javabin&waitSearcher=true&softCommit=false} {commit=} 0 2
   [junit4]   2> 1584674 T6236 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/127.0.0.1,port=46220,localport=44949], receiveBufferSize:131015
   [junit4]   2> 1584675 T6236 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=40851,localport=54498], receiveBufferSize=131003
   [junit4]   2> 1584677 T6236 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/127.0.0.1,port=46222,localport=44949], receiveBufferSize:131015
   [junit4]   2> 1584678 T6236 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=40851,localport=54500], receiveBufferSize=131003
   [junit4]   2> ASYNC  NEW_CORE C1397 name=collection1 org.apache.solr.core.SolrCore@4dd4cf8e url=http://127.0.0.1:44949/collection1 node=127.0.0.1:44949_ C1397_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:44949, core=collection1, node_name=127.0.0.1:44949_, leader=true}
   [junit4]   2> 1584679 T6223 C1397 P44949 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1584679 T6223 C1397 P44949 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1584679 T6223 C1397 P44949 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1584679 T6223 C1397 P44949 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1584680 T6223 C1397 P44949 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={commit=true&commit_end_point=true&expungeDeletes=false&version=2&distrib.from=http://127.0.0.1:44949/collection1/&update.distrib=FROMLEADER&wt=javabin&waitSearcher=true&softCommit=false&openSearcher=true} {commit=} 0 2
   [junit4]   2> 1584681 T6255 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/127.0.0.1,port=44016,localport=48572], receiveBufferSize:131015
   [junit4]   2> 1584681 T6255 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=58625,localport=41081], receiveBufferSize=131003
   [junit4]   2>  C1395_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:48572, core=collection1, node_name=127.0.0.1:48572_, leader=true}
   [junit4]   2> 1584682 T6243 C1395 P48572 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1584682 T6243 C1395 P48572 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1584682 T6243 C1395 P48572 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1584683 T6243

[...truncated too long message...]

.timezone=Asia/Aqtau -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    118s J1 | HttpPartitionTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: KeeperErrorCode = NodeExists for /overseer/collection-queue-work/qnr-0000000002
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([45878A30BD887025:C4610428CAD71019]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1508)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1468)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1527)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1464)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.testRf3(HttpPartitionTest.java:223)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:145)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1675063 T6185 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-45878A30BD887025-001
   [junit4]   2> 118420 T6184 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 5 leaked thread(s).
   [junit4]   2> 1675744 T6207 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1675841 T6343 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 1675841 T6230 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1675925 T6347 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=ar_BH, timezone=Asia/Aqtau
   [junit4]   2> NOTE: Linux 3.8.0-39-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=48390064,total=256901120
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrXmlPersistence, TestNRTOpen, DirectSolrConnectionTest, TestWordDelimiterFilterFactory, TestRangeQuery, CachingDirectoryFactoryTest, PluginInfoTest, SuggesterTSTTest, HdfsWriteToMultipleCollectionsTest, StandardRequestHandlerTest, TestSolrXml, TestAtomicUpdateErrorCases, TestSimpleQParserPlugin, TestManagedSynonymFilterFactory, StressHdfsTest, BasicDistributedZk2Test, TestPhraseSuggestions, TestBinaryResponseWriter, TestUpdate, TestSchemaVersionResource, TestCloudManagedSchema, NotRequiredUniqueKeyTest, TestRecoveryHdfs, TestFoldingMultitermQuery, SolrIndexConfigTest, TestZkChroot, DefaultValueUpdateProcessorTest, TestShortCircuitedRequests, TestNumberUtils, ZkSolrClientTest, TestIndexSearcher, TestArbitraryIndexDir, SignatureUpdateProcessorFactoryTest, XsltUpdateRequestHandlerTest, TestRandomMergePolicy, TestAnalyzeInfixSuggestions, TestQueryTypes, TestFuzzyAnalyzedSuggestions, TestMiniSolrCloudCluster, IndexBasedSpellCheckerTest, ConnectionManagerTest, TestJmxIntegration, TestCSVResponseWriter, TestCustomSort, TestHashPartitioner, TestDefaultSimilarityFactory, TestQuerySenderNoQuery, TestSweetSpotSimilarityFactory, AliasIntegrationTest, TestExceedMaxTermLength, TestBinaryField, TestManagedSchema, TestFastLRUCache, TestShardHandlerFactory, ZkNodePropsTest, TestFieldTypeResource, TestSolrQueryParserResource, StatsComponentTest, TestRestManager, RequestHandlersTest, TestUtils, ShardSplitTest, TestCloudManagedSchemaAddField, TestSolrCoreProperties, SimplePostToolTest, RAMDirectoryFactoryTest, TestDistributedSearch, TestMultiCoreConfBootstrap, CursorPagingTest, DebugComponentTest, PreAnalyzedFieldTest, TestRandomFaceting, SortByFunctionTest, TestCollationField, DocValuesMultiTest, CollectionsAPIAsyncDistributedZkTest, TestCollationKeyRangeQueries, TermVectorComponentDistributedTest, CurrencyFieldOpenExchangeTest, TestRTGBase, TestCSVLoader, TestXIncludeConfig, TestDynamicFieldCollectionResource, HdfsSyncSliceTest, IndexSchemaRuntimeFieldTest, SpellCheckCollatorTest, MultiTermTest, TestInitQParser, TestFastOutputStream, TriLevelCompositeIdRoutingTest, TestConfigSets, DirectUpdateHandlerTest, HighlighterConfigTest, CollectionsAPIDistributedZkTest, NumericFieldsTest, TestFastWriter, LeaderElectionIntegrationTest, RecoveryZkTest, TestReplicationHandler, ShardRoutingCustomTest, TestDistributedGrouping, ZkControllerTest, TestStressReorder, TestReload, TestStressVersions, TestSolr4Spatial, SolrCmdDistributorTest, QueryElevationComponentTest, TestSort, TestFunctionQuery, TestLazyCores, TestBadConfig, OverseerCollectionProcessorTest, HighlighterTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SolrRequestParserTest, SpatialFilterTest, SuggesterWFSTTest, NoCacheHeaderTest, SolrCoreCheckLockOnStartupTest, StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, CacheHeaderTest, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestValueSourceCache, TermVectorComponentTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, SolrPluginUtilsTest, ReturnFieldsTest, MBeansHandlerTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, SearchHandlerTest, AlternateDirectoryTest, BadComponentTest, TestStressRecovery, SpellPossibilityIteratorTest, TestCodecSupport, EchoParamsTest, TestPerFieldSimilarity, TestIBSimilarityFactory, TestJmxMonitoredMap, ResourceLoaderTest, ChaosMonkeyNothingIsSafeTest, DeleteReplicaTest, DistribCursorPagingTest, HttpPartitionTest]
   [junit4] Completed on J1 in 119.45s, 1 test, 1 error <<< FAILURES!

[...truncated 192 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 error, 36 ignored (15 assumptions)

Total time: 66 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure