You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/04/17 13:42:18 UTC

[JENKINS] Lucene-Solr-4.8-Linux (64bit/jdk1.8.0_20-ea-b05) - Build # 18 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.8-Linux/18/
Java: 64bit/jdk1.8.0_20-ea-b05 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerStatusTest.testDistribSearch

Error Message:
overseerstatus the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: overseerstatus the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([77536D3E5BE4AF:8191DD7549048493]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.OverseerStatusTest.invokeCollectionApi(OverseerStatusTest.java:121)
	at org.apache.solr.cloud.OverseerStatusTest.doTest(OverseerStatusTest.java:96)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.cloud.OverseerTest.testShardAssignmentBigger

Error Message:
Unable to verify all cores have been returned an id expected:<154> but was:<153>

Stack Trace:
java.lang.AssertionError: Unable to verify all cores have been returned an id expected:<154> but was:<153>
	at __randomizedtesting.SeedInfo.seed([77536D3E5BE4AF:9399AD16D4321BCC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.OverseerTest.testShardAssignmentBigger(OverseerTest.java:432)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 11005 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerStatusTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/init-core-data-001
   [junit4]   2> 363964 T1811 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 363967 T1811 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 363968 T1811 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 363969 T1812 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 364069 T1811 oasc.ZkTestServer.run start zk server on port:58898
   [junit4]   2> 364070 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 364145 T1818 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a04f8c5 name:ZooKeeperConnection Watcher:127.0.0.1:58898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 364145 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 364146 T1811 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 364148 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 364150 T1820 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f49421 name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 364150 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 364150 T1811 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 364151 T1811 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 364153 T1811 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 364154 T1811 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 364155 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 364156 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 364158 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 364159 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 364160 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 364161 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 364163 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 364164 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 364165 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 364166 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 364167 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 364167 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 364173 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 364173 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 364175 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 364175 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 364177 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 364177 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 364179 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 364179 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 364182 T1811 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 364182 T1811 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 364414 T1811 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 364417 T1811 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43229
   [junit4]   2> 364418 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 364418 T1811 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 364419 T1811 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002
   [junit4]   2> 364419 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/'
   [junit4]   2> 364430 T1811 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/solr.xml
   [junit4]   2> 364439 T1811 oasc.CoreContainer.<init> New CoreContainer 2085931637
   [junit4]   2> 364440 T1811 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/]
   [junit4]   2> 364440 T1811 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 364440 T1811 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 364440 T1811 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 364441 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 364441 T1811 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 364441 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 364441 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 364441 T1811 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 364442 T1811 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 364442 T1811 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 364442 T1811 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 364443 T1811 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 364443 T1811 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58898/solr
   [junit4]   2> 364443 T1811 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 364444 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 364445 T1831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c6346da name:ZooKeeperConnection Watcher:127.0.0.1:58898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 364445 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 364447 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 364448 T1833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6df331 name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 364448 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 364449 T1811 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 364451 T1811 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 364454 T1811 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 364456 T1811 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 364458 T1811 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 364459 T1811 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 364460 T1811 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43229_
   [junit4]   2> 364461 T1811 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43229_
   [junit4]   2> 364462 T1811 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 364463 T1811 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 364465 T1811 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43229_
   [junit4]   2> 364465 T1811 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 364466 T1811 oasc.Overseer.start Overseer (id=91601862394970115-127.0.0.1:43229_-n_0000000000) starting
   [junit4]   2> 364472 T1811 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 364478 T1835 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 364478 T1811 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 364478 T1835 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:43229_
   [junit4]   2> 364479 T1811 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 364480 T1811 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 364482 T1834 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 364483 T1836 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 364483 T1836 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 364484 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 364484 T1836 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 364485 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 364485 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43229",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43229_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 364486 T1834 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 364486 T1834 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 364487 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 364488 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 365484 T1836 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 365485 T1836 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 365485 T1836 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 365485 T1836 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 365486 T1836 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 365486 T1836 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/collection1/'
   [junit4]   2> 365487 T1836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 365487 T1836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 365487 T1836 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 365516 T1836 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 365527 T1836 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 365528 T1836 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 365534 T1836 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 365658 T1836 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 365660 T1836 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 365661 T1836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 365663 T1836 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 365678 T1836 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 365678 T1836 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 365678 T1836 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-002/collection1/, dataDir=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/control/data/
   [junit4]   2> 365678 T1836 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ca587ec
   [junit4]   2> 365679 T1836 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/control/data
   [junit4]   2> 365679 T1836 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/control/data/index/
   [junit4]   2> 365680 T1836 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 365680 T1836 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/control/data/index
   [junit4]   2> 365680 T1836 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=59.6162109375, floorSegmentMB=1.5595703125, forceMergeDeletesPctAllowed=13.336408103198599, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 365681 T1836 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@699cea9e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d529ada),segFN=segments_1,generation=1}
   [junit4]   2> 365681 T1836 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 365683 T1836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 365683 T1836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 365683 T1836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 365683 T1836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 365684 T1836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 365684 T1836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 365684 T1836 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 365684 T1836 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 365684 T1836 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 365684 T1836 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 365685 T1836 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 365685 T1836 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 365685 T1836 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 365685 T1836 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 365685 T1836 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 365686 T1836 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 365688 T1836 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 365689 T1836 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 365689 T1836 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 365690 T1836 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=52.89453125, floorSegmentMB=0.3173828125, forceMergeDeletesPctAllowed=1.5428595206039009, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 365690 T1836 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@699cea9e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4d529ada),segFN=segments_1,generation=1}
   [junit4]   2> 365690 T1836 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 365690 T1836 oass.SolrIndexSearcher.<init> Opening Searcher@47a2b034[collection1] main
   [junit4]   2> 365691 T1836 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 365691 T1836 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 365692 T1836 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 365692 T1836 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 365692 T1836 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 365692 T1836 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 365693 T1836 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 365693 T1836 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 365693 T1836 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 365693 T1836 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 365694 T1837 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47a2b034[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 365695 T1836 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 365695 T1840 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43229 collection:control_collection shard:shard1
   [junit4]   2> 365696 T1811 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 365696 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 365696 T1840 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 365697 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 365698 T1842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46df75fd name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 365698 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 365699 T1811 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 365700 T1840 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 365700 T1811 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 365701 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365701 T1840 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 365702 T1840 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C792 name=collection1 org.apache.solr.core.SolrCore@6a1522a1 url=http://127.0.0.1:43229/collection1 node=127.0.0.1:43229_ C792_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:43229, core=collection1, node_name=127.0.0.1:43229_}
   [junit4]   2> 365702 T1840 C792 P43229 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43229/collection1/
   [junit4]   2> 365702 T1840 C792 P43229 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 365702 T1840 C792 P43229 oasc.SyncStrategy.syncToMe http://127.0.0.1:43229/collection1/ has no replicas
   [junit4]   2> 365702 T1840 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43229/collection1/ shard1
   [junit4]   2> 365702 T1840 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 365703 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 365705 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365710 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365712 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365814 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 365814 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 365861 T1840 oasc.ZkController.register We are http://127.0.0.1:43229/collection1/ and leader is http://127.0.0.1:43229/collection1/
   [junit4]   2> 365861 T1840 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43229
   [junit4]   2> 365861 T1840 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 365861 T1840 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 365861 T1840 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 365862 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365862 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365863 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365863 T1840 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 365864 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 365864 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43229",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43229_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 365866 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 365931 T1811 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 365932 T1811 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 365934 T1811 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52809
   [junit4]   2> 365934 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 365935 T1811 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 365935 T1811 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003
   [junit4]   2> 365935 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/'
   [junit4]   2> 365951 T1811 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/solr.xml
   [junit4]   2> 365964 T1811 oasc.CoreContainer.<init> New CoreContainer 715327688
   [junit4]   2> 365964 T1811 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/]
   [junit4]   2> 365965 T1811 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 365965 T1811 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 365966 T1811 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 365966 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 365966 T1811 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 365967 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 365967 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 365967 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 365967 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 365968 T1811 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 365968 T1811 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 365969 T1811 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 365969 T1811 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 365970 T1811 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 365970 T1811 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58898/solr
   [junit4]   2> 365971 T1811 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 365971 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 365973 T1853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2eb614eb name:ZooKeeperConnection Watcher:127.0.0.1:58898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 365973 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 365978 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 365979 T1855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4efd6b3a name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 365979 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 365983 T1811 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 366986 T1811 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52809_
   [junit4]   2> 366987 T1811 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52809_
   [junit4]   2> 366990 T1833 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 366990 T1842 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 366990 T1855 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 366994 T1856 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 366995 T1856 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 366996 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 366996 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 366996 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 366996 T1856 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 366997 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 366998 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52809",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52809_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 366998 T1834 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 366998 T1834 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 367001 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 367104 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 367104 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 367104 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 367997 T1856 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 367997 T1856 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 367998 T1856 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 367998 T1856 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 367998 T1856 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 367999 T1856 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/collection1/'
   [junit4]   2> 367999 T1856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 368000 T1856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 368000 T1856 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 368037 T1856 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 368050 T1856 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 368052 T1856 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 368057 T1856 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 368195 T1856 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 368196 T1856 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 368198 T1856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 368203 T1856 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 368222 T1856 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 368223 T1856 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 368223 T1856 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-003/collection1/, dataDir=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty1/
   [junit4]   2> 368223 T1856 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ca587ec
   [junit4]   2> 368224 T1856 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty1
   [junit4]   2> 368225 T1856 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty1/index/
   [junit4]   2> 368225 T1856 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 368225 T1856 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty1/index
   [junit4]   2> 368226 T1856 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=59.6162109375, floorSegmentMB=1.5595703125, forceMergeDeletesPctAllowed=13.336408103198599, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 368226 T1856 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1a100472 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@42156e87),segFN=segments_1,generation=1}
   [junit4]   2> 368227 T1856 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 368230 T1856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 368230 T1856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 368231 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 368231 T1856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 368231 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 368231 T1856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 368232 T1856 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 368232 T1856 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 368232 T1856 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 368233 T1856 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 368233 T1856 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 368233 T1856 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 368234 T1856 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 368234 T1856 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 368234 T1856 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 368235 T1856 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 368237 T1856 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 368238 T1856 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 368239 T1856 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 368239 T1856 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=52.89453125, floorSegmentMB=0.3173828125, forceMergeDeletesPctAllowed=1.5428595206039009, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 368240 T1856 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1a100472 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@42156e87),segFN=segments_1,generation=1}
   [junit4]   2> 368240 T1856 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 368240 T1856 oass.SolrIndexSearcher.<init> Opening Searcher@e810e44[collection1] main
   [junit4]   2> 368241 T1856 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 368242 T1856 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 368242 T1856 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 368242 T1856 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 368242 T1856 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 368243 T1856 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 368243 T1856 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 368243 T1856 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 368244 T1856 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 368244 T1856 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 368246 T1857 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e810e44[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 368248 T1856 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 368249 T1860 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52809 collection:collection1 shard:shard2
   [junit4]   2> 368250 T1811 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 368250 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 368250 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 368257 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 368258 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368259 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368259 T1860 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 368259 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368259 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 368260 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C793 name=collection1 org.apache.solr.core.SolrCore@17819e13 url=http://127.0.0.1:52809/collection1 node=127.0.0.1:52809_ C793_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52809, core=collection1, node_name=127.0.0.1:52809_}
   [junit4]   2> 368260 T1860 C793 P52809 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52809/collection1/
   [junit4]   2> 368261 T1860 C793 P52809 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 368261 T1860 C793 P52809 oasc.SyncStrategy.syncToMe http://127.0.0.1:52809/collection1/ has no replicas
   [junit4]   2> 368261 T1860 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52809/collection1/ shard2
   [junit4]   2> 368262 T1860 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 368263 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368269 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368271 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368377 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368377 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368377 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368419 T1860 oasc.ZkController.register We are http://127.0.0.1:52809/collection1/ and leader is http://127.0.0.1:52809/collection1/
   [junit4]   2> 368420 T1860 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52809
   [junit4]   2> 368420 T1860 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 368420 T1860 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 368420 T1860 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 368421 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368421 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368422 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368422 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 368423 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52809",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52809_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 368423 T1860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 368426 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 368497 T1811 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 368498 T1811 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 368500 T1811 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35240
   [junit4]   2> 368500 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 368500 T1811 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 368501 T1811 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004
   [junit4]   2> 368501 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/'
   [junit4]   2> 368514 T1811 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/solr.xml
   [junit4]   2> 368527 T1811 oasc.CoreContainer.<init> New CoreContainer 356111198
   [junit4]   2> 368527 T1811 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/]
   [junit4]   2> 368527 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368527 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368528 T1811 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 368528 T1811 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 368527 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 368528 T1811 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 368529 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 368529 T1811 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 368530 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 368530 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 368530 T1811 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 368531 T1811 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 368531 T1811 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 368532 T1811 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 368532 T1811 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 368533 T1811 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58898/solr
   [junit4]   2> 368533 T1811 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 368534 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 368535 T1871 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d09f0bc name:ZooKeeperConnection Watcher:127.0.0.1:58898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 368535 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 368537 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 368538 T1873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27a1bf9c name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 368538 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 368542 T1811 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 369544 T1811 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35240_
   [junit4]   2> 369546 T1811 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35240_
   [junit4]   2> 369550 T1833 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 369550 T1842 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 369550 T1873 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 369550 T1855 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 369556 T1874 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 369556 T1874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 369557 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 369558 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 369558 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 369557 T1874 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 369560 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 369560 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35240",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35240_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 369561 T1834 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 369561 T1834 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 369563 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 369665 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 369665 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 369665 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 369665 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 370558 T1874 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 370559 T1874 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 370559 T1874 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 370559 T1874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 370560 T1874 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 370560 T1874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/collection1/'
   [junit4]   2> 370561 T1874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 370561 T1874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 370562 T1874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 370587 T1874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 370598 T1874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 370599 T1874 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 370604 T1874 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 370728 T1874 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 370730 T1874 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 370731 T1874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 370734 T1874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 370746 T1874 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 370747 T1874 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 370747 T1874 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-004/collection1/, dataDir=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty2/
   [junit4]   2> 370747 T1874 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ca587ec
   [junit4]   2> 370749 T1874 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty2
   [junit4]   2> 370749 T1874 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty2/index/
   [junit4]   2> 370750 T1874 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 370750 T1874 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-001/jetty2/index
   [junit4]   2> 370750 T1874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=59.6162109375, floorSegmentMB=1.5595703125, forceMergeDeletesPctAllowed=13.336408103198599, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 370751 T1874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@663acda9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7fdcc66f),segFN=segments_1,generation=1}
   [junit4]   2> 370751 T1874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 370754 T1874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 370754 T1874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 370755 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 370755 T1874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 370755 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 370755 T1874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 370756 T1874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 370756 T1874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 370756 T1874 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 370757 T1874 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 370757 T1874 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 370757 T1874 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 370758 T1874 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 370758 T1874 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 370758 T1874 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 370759 T1874 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 370761 T1874 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 370763 T1874 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 370763 T1874 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 370763 T1874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=52.89453125, floorSegmentMB=0.3173828125, forceMergeDeletesPctAllowed=1.5428595206039009, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 370764 T1874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@663acda9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7fdcc66f),segFN=segments_1,generation=1}
   [junit4]   2> 370764 T1874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 370764 T1874 oass.SolrIndexSearcher.<init> Opening Searcher@1cfab422[collection1] main
   [junit4]   2> 370764 T1874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 370765 T1874 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 370765 T1874 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 370765 T1874 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 370766 T1874 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 370766 T1874 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 370766 T1874 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 370766 T1874 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 370766 T1874 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 370767 T1874 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 370768 T1875 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1cfab422[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 370770 T1874 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 370770 T1878 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35240 collection:collection1 shard:shard1
   [junit4]   2> 370771 T1811 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 370771 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 370773 T1878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 370777 T1878 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 370778 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370778 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370778 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370779 T1878 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 370779 T1878 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 370779 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C794 name=collection1 org.apache.solr.core.SolrCore@97a68b2 url=http://127.0.0.1:35240/collection1 node=127.0.0.1:35240_ C794_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35240, core=collection1, node_name=127.0.0.1:35240_}
   [junit4]   2> 370779 T1878 C794 P35240 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35240/collection1/
   [junit4]   2> 370779 T1878 C794 P35240 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 370780 T1878 C794 P35240 oasc.SyncStrategy.syncToMe http://127.0.0.1:35240/collection1/ has no replicas
   [junit4]   2> 370780 T1878 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35240/collection1/ shard1
   [junit4]   2> 370780 T1878 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 370781 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370783 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370785 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370891 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 370892 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 370892 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 370892 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 370934 T1878 oasc.ZkController.register We are http://127.0.0.1:35240/collection1/ and leader is http://127.0.0.1:35240/collection1/
   [junit4]   2> 370934 T1878 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35240
   [junit4]   2> 370934 T1878 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 370934 T1878 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 370935 T1878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 370935 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370936 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370936 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370936 T1878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 370937 T1834 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 370937 T1834 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35240",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35240_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 370939 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 371016 T1811 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 371017 T1811 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 371019 T1811 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45362
   [junit4]   2> 371019 T1811 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 371019 T1811 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 371019 T1811 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-005
   [junit4]   2> 371020 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-005/'
   [junit4]   2> 371030 T1811 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-005/solr.xml
   [junit4]   2> 371038 T1811 oasc.CoreContainer.<init> New CoreContainer 1401475019
   [junit4]   2> 371039 T1811 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-77536D3E5BE4AF-001/tempDir-005/]
   [junit4]   2> 371039 T1811 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 371039 T1811 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 371040 T1811 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 371040 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 371040 T1811 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 371040 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 371040 T1855 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 371040 T1842 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 371040 T1833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 371040 T1873 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 371041 T1811 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 371042 T1811 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 371042 T1811 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 371043 T1811 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 371043 T1811 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 371043 T1811 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 371043 T1811 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58898/solr
   [junit4]   2> 371044 T1811 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 371044 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 371045 T1889 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@945e9ef name:ZooKeeperConnection Watcher:127.0.0.1:58898 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 371045 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 371047 T1811 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 371048 T1891 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@124980ef name:ZooKeeperConnection Watcher:127.0.0.1:58898/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 371048 T1811 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 371053 T1811 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 372055 T1811 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45362_
   [junit4]   2> 372056 T1811 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45362_
   [junit4]   2> 372057 T1842 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 372057 T1873 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 372058 T1891 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 372058 T1833 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 372057 T1855 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 372061 T1892 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 372061 T1892 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 372061 T1833 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37

[...truncated too long message...]

973 T3548 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1300973 T3548 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1300974 T3548 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1300976 T3548 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48065_solr
   [junit4]   2> 1300977 T3548 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1300978 T3548 oasc.Overseer.start Overseer (id=91601923793616900-127.0.0.1:48065_solr-n_0000000000) starting
   [junit4]   2> 1300979 T3548 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1300981 T3548 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1300983 T3548 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1300985 T3548 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1300986 T3548 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1300988 T3548 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1300993 T4034 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1300994 T4033 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1300994 T4034 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:48065_solr
   [junit4]   2> 1300997 T4033 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1300997 T4033 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1300997 T4033 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 1300997 T4033 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1300999 T4032 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1301000 T4024 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1301000 T4030 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1301494 T3548 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1301502 T3548 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1301506 T4032 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1301506 T3548 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1301507 T4033 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1301509 T4024 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1301509 T4030 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1301513 T3548 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48065 48065
   [junit4]   2> 1301804 T4018 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48065 48065
   [junit4]   2> 1301805 T3548 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   2> 1301805 T4033 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48065_solr
   [junit4]   2> 1301805 T4035 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1301806 T3548 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1301806 T3548 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1304807 T3548 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-77536D3E5BE4AF-001
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ga, timezone=Africa/Abidjan
   [junit4]   2> NOTE: Linux 3.8.0-38-generic amd64/Oracle Corporation 1.8.0_20-ea (64-bit)/cpus=8,threads=1,free=120882656,total=411041792
   [junit4]   2> NOTE: All tests run in this JVM: [DocValuesTest, BasicFunctionalityTest, UpdateRequestProcessorFactoryTest, TestNonNRTOpen, TestFieldCollectionResource, TestFieldSortValues, TestBlendedInfixSuggestions, TestRandomDVFaceting, TestRecoveryHdfs, SystemInfoHandlerTest, BasicDistributedZkTest, TermsComponentTest, TestCloudManagedSchemaAddField, TestTrie, TestRealTimeGet, TestLRUCache, ResponseLogComponentTest, DistributedQueryComponentOptimizationTest, TestReplicationHandler, DocumentBuilderTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolrQueryParserResource, DocValuesMissingTest, StatelessScriptUpdateProcessorFactoryTest, ShardRoutingTest, OverseerCollectionProcessorTest, TestPHPSerializedResponseWriter, EchoParamsTest, QueryElevationComponentTest, SolrInfoMBeanTest, OverseerStatusTest, TestReloadAndDeleteDocs, CoreAdminRequestStatusTest, TestRestManager, TestShardHandlerFactory, FileBasedSpellCheckerTest, ZkControllerTest, TestLMDirichletSimilarityFactory, TestDefaultSearchFieldResource, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestLMJelinekMercerSimilarityFactory, TestSuggestSpellingConverter, TestUniqueKeyFieldResource, TestBadConfig, TestPostingsSolrHighlighter, SchemaVersionSpecificBehaviorTest, TestClassNameShortening, TestDocumentBuilder, TestConfig, UpdateParamsTest, MoreLikeThisHandlerTest, TestJoin, TestSystemIdResolver, HdfsRecoveryZkTest, TestSimpleQParserPlugin, TestGroupingSearch, DistributedQueryComponentCustomSortTest, TestSort, TestConfigSets, TestLuceneMatchVersion, PrimitiveFieldTypeTest, TestSolrXMLSerializer, TestFieldResource, CacheHeaderTest, TestFieldTypeResource, IndexSchemaTest, TestExtendedDismaxParser, UUIDFieldTest, SolrTestCaseJ4Test, PolyFieldTest, TestManagedResourceStorage, CoreMergeIndexesAdminHandlerTest, TestShortCircuitedRequests, TestExceedMaxTermLength, FieldMutatingUpdateProcessorTest, TestRequestStatusCollectionAPI, TestRandomMergePolicy, SoftAutoCommitTest, HdfsUnloadDistributedZkTest, TestLFUCache, DateMathParserTest, TermVectorComponentTest, TriLevelCompositeIdRoutingTest, TestRemoteStreaming, BadComponentTest, ClusterStateUpdateTest, ShowFileRequestHandlerTest, LeaderElectionTest, ExternalFileFieldSortTest, TestComplexPhraseQParserPlugin, ScriptEngineTest, QueryParsingTest, TestJmxMonitoredMap, OverseerTest]
   [junit4] Completed on J0 in 348.61s, 9 tests, 1 failure <<< FAILURES!

[...truncated 582 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:1282: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:908: There were test failures: 387 suites, 1619 tests, 1 error, 1 failure, 37 ignored (16 assumptions)

Total time: 66 minutes 51 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_20-ea-b05 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure