You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/17 13:34:49 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9127 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9127/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.MigrateRouteKeyTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest:     1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest: 
   1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([C42072D286CD121B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.MigrateRouteKeyTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([C42072D286CD121B]:0)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
reloadcollection the collection time out:60s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reloadcollection the collection time out:60s
	at __randomizedtesting.SeedInfo.seed([C42072D286CD121B:45C6FCCAF1927227]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:765)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:196)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10411 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 495353 T2112 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 495356 T2112 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1389961084578
   [junit4]   2> 495357 T2112 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 495357 T2113 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 495457 T2112 oasc.ZkTestServer.run start zk server on port:36917
   [junit4]   2> 495459 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495462 T2119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f891a name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495462 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495462 T2112 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 495467 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495468 T2121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c0cf48 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495468 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495468 T2112 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 495471 T2112 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 495473 T2112 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 495475 T2112 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 495476 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 495477 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 495480 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 495481 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 495484 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 495484 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 495488 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 495489 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 495492 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 495493 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 495496 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 495497 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 495499 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 495499 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 495502 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 495502 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 495504 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 495505 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 495508 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 495508 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 495512 T2112 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 495512 T2112 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 495600 T2112 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 495602 T2112 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46361
   [junit4]   2> 495603 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 495603 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 495603 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737
   [junit4]   2> 495604 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737/'
   [junit4]   2> 495617 T2112 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737/solr.xml
   [junit4]   2> 495657 T2112 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 495657 T2112 oasc.CoreContainer.<init> New CoreContainer 1095485
   [junit4]   2> 495657 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737/]
   [junit4]   2> 495659 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 495659 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 495659 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 495659 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 495660 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 495660 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 495660 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 495661 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 495661 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 495663 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 495663 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 495663 T2112 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 495664 T2112 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36917/solr
   [junit4]   2> 495664 T2112 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 495665 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495667 T2132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16e1b5e name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495668 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495671 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495672 T2134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e495c1 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495673 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495674 T2112 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 495678 T2112 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 495684 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 495687 T2112 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46361_
   [junit4]   2> 495689 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46361_
   [junit4]   2> 495693 T2112 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 495697 T2112 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 495702 T2112 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 495706 T2112 oasc.Overseer.start Overseer (id=91092489638969347-127.0.0.1:46361_-n_0000000000) starting
   [junit4]   2> 495711 T2112 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 495718 T2136 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 495719 T2112 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 495719 T2136 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 495723 T2112 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 495727 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 495732 T2135 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 495736 T2112 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737
   [junit4]   2> 495738 T2112 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1389961084737/conf/
   [junit4]   2> 495774 T2112 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 495775 T2112 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 495776 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 495777 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495778 T2139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d53f2 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495778 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495779 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 495781 T2112 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 495846 T2112 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 495848 T2112 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55739
   [junit4]   2> 495849 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 495849 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 495849 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003
   [junit4]   2> 495850 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/'
   [junit4]   2> 495862 T2112 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solr.xml
   [junit4]   2> 495902 T2112 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 495902 T2112 oasc.CoreContainer.<init> New CoreContainer 7959646
   [junit4]   2> 495903 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/]
   [junit4]   2> 495904 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 495904 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 495905 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 495905 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 495905 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 495905 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 495906 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 495906 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 495906 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 495908 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 495909 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 495909 T2112 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 495909 T2112 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36917/solr
   [junit4]   2> 495909 T2112 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 495910 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495912 T2150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c6e135 name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495913 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495917 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 495918 T2152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15948a9 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 495919 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 495925 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 496931 T2112 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55739_
   [junit4]   2> 496934 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55739_
   [junit4]   2> 496938 T2134 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 496938 T2139 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 496938 T2152 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 496942 T2112 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003
   [junit4]   2> 496943 T2112 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/conf/
   [junit4]   2> 496974 T2112 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 496975 T2112 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 496975 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 497045 T2112 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 497047 T2112 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41810
   [junit4]   2> 497047 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 497048 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 497048 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197
   [junit4]   2> 497048 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/'
   [junit4]   2> 497061 T2112 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solr.xml
   [junit4]   2> 497102 T2112 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 497103 T2112 oasc.CoreContainer.<init> New CoreContainer 17151249
   [junit4]   2> 497103 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/]
   [junit4]   2> 497104 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 497105 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 497105 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 497105 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 497106 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 497106 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 497106 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 497107 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 497107 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 497109 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 497109 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 497110 T2112 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 497110 T2112 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36917/solr
   [junit4]   2> 497110 T2112 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 497111 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 497121 T2164 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b29177 name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 497121 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 497124 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 497125 T2166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@611920 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 497125 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 497128 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 498131 T2112 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41810_
   [junit4]   2> 498133 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41810_
   [junit4]   2> 498138 T2152 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 498138 T2134 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 498139 T2166 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 498138 T2139 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 498149 T2112 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197
   [junit4]   2> 498151 T2112 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/conf/
   [junit4]   2> 498185 T2112 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 498186 T2112 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 498186 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 498253 T2112 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 498267 T2112 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41136
   [junit4]   2> 498268 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 498269 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 498269 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408
   [junit4]   2> 498269 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/'
   [junit4]   2> 498291 T2112 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solr.xml
   [junit4]   2> 498334 T2112 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 498335 T2112 oasc.CoreContainer.<init> New CoreContainer 23130354
   [junit4]   2> 498335 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/]
   [junit4]   2> 498336 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 498336 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 498337 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 498337 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 498337 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 498338 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 498338 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 498338 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 498338 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 498341 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 498341 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 498341 T2112 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 498341 T2112 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36917/solr
   [junit4]   2> 498342 T2112 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 498342 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 498344 T2178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16ac26d name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 498344 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 498346 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 498347 T2180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e01a42 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 498348 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 498351 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 499354 T2112 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41136_
   [junit4]   2> 499356 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41136_
   [junit4]   2> 499358 T2152 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 499359 T2134 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 499359 T2139 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 499359 T2180 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 499359 T2166 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 499364 T2112 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408
   [junit4]   2> 499365 T2112 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/conf/
   [junit4]   2> 499389 T2112 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 499390 T2112 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 499390 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 499456 T2112 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 499459 T2112 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37677
   [junit4]   2> 499460 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 499461 T2112 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 499461 T2112 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611
   [junit4]   2> 499462 T2112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/'
   [junit4]   2> 499481 T2112 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solr.xml
   [junit4]   2> 499531 T2112 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 499532 T2112 oasc.CoreContainer.<init> New CoreContainer 10859344
   [junit4]   2> 499532 T2112 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/]
   [junit4]   2> 499533 T2112 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 499533 T2112 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 499534 T2112 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 499534 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 499534 T2112 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 499534 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 499535 T2112 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 499535 T2112 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 499535 T2112 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 499537 T2112 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 499538 T2112 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 499538 T2112 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 499538 T2112 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36917/solr
   [junit4]   2> 499539 T2112 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 499539 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499541 T2192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9f53c7 name:ZooKeeperConnection Watcher:127.0.0.1:36917 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499541 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499543 T2112 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499544 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84c446 name:ZooKeeperConnection Watcher:127.0.0.1:36917/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499544 T2112 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499547 T2112 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 500550 T2112 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37677_
   [junit4]   2> 500552 T2112 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37677_
   [junit4]   2> 500558 T2152 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500558 T2166 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500559 T2194 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500559 T2139 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500559 T2134 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500559 T2180 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 500570 T2112 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611
   [junit4]   2> 500571 T2112 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/conf/
   [junit4]   2> 500598 T2112 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 500599 T2112 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 500599 T2112 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 500604 T2146 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&router.field=myOwnField&router.name=compositeId&name=solrj_collection&wt=javabin&numShards=2&action=CREATE&version=2&collection.configName=conf1
   [junit4]   2> 500606 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500607 T2136 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 500608 T2136 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 500608 T2136 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 500609 T2136 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 500614 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500616 T2135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 500618 T2135 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 500618 T2135 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 500622 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500623 T2166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500624 T2152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500624 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500623 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500623 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500623 T2180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500715 T2136 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 500715 T2136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:41136_
   [junit4]   2> 500715 T2136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:55739_
   [junit4]   2> 500716 T2136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:41810_
   [junit4]   2> 500717 T2136 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:37677_
   [junit4]   2> 500718 T2176 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 500718 T2161 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 500718 T2148 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 500719 T2187 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 500719 T2176 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 500719 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500719 T2161 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 500720 T2187 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 500720 T2148 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 500721 T2135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 500722 T2135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41136",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41136_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 500724 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500724 T2135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41810",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41810_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 500726 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500727 T2135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55739",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55739_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 500729 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500730 T2135 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37677",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37677_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 500734 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 500836 T2152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500836 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500836 T2166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500836 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500836 T2180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 500837 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 501720 T2176 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 501720 T2161 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 501721 T2187 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 501721 T2148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 501721 T2176 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1
   [junit4]   2> 501721 T2176 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 501721 T2187 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2
   [junit4]   2> 501722 T2187 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 501721 T2161 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1
   [junit4]   2> 501723 T2148 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2
   [junit4]   2> 501723 T2148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 501723 T2187 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 501724 T2187 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 501722 T2176 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 501724 T2148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 501725 T2148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 501723 T2161 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 501725 T2187 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/'
   [junit4]   2> 501724 T2176 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 501726 T2161 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 501726 T2161 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 501727 T2176 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/'
   [junit4]   2> 501726 T2148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/'
   [junit4]   2> 501728 T2161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/'
   [junit4]   2> 501780 T2148 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 501785 T2187 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 501786 T2176 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 501810 T2161 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 501840 T2148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 501842 T2148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 501851 T2148 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 501864 T2161 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 501865 T2161 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 501871 T2187 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 501872 T2187 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 501873 T2176 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 501875 T2176 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 501875 T2161 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 501886 T2187 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 501895 T2176 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 502419 T2148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 502429 T2148 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 502434 T2148 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 502443 T2161 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 502448 T2148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502450 T2161 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 502451 T2148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502453 T2161 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 502455 T2148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502456 T2148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502457 T2148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502458 T2148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502459 T2148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502459 T2148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502459 T2148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 502460 T2148 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/data/
   [junit4]   2> 502460 T2148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@134f867
   [junit4]   2> 502467 T2148 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/data
   [junit4]   2> 502468 T2148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 502468 T2148 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 502469 T2148 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1389961085003/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 502470 T2148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5037992432018531]
   [junit4]   2> 502470 T2148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1235ce5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@93791f),segFN=segments_1,generation=1}
   [junit4]   2> 502471 T2148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502472 T2161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502476 T2148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 502476 T2148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 502477 T2148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 502477 T2148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502477 T2161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502478 T2148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502478 T2148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 502479 T2148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502480 T2148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502480 T2148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 502481 T2148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 502482 T2148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 502482 T2148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 502483 T2148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 502483 T2161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502483 T2148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 502484 T2148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 502484 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502485 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502485 T2148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 502485 T2161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502486 T2148 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 502486 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502487 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502487 T2161 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 502488 T2161 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/data/
   [junit4]   2> 502488 T2161 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@134f867
   [junit4]   2> 502489 T2161 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/data
   [junit4]   2> 502489 T2148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 502490 T2161 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 502490 T2161 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 502491 T2161 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1389961086197/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 502492 T2161 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5037992432018531]
   [junit4]   2> 502492 T2161 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1d740c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@72c648),segFN=segments_1,generation=1}
   [junit4]   2> 502493 T2161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502494 T2148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 502494 T2148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 502495 T2187 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 502497 T2148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=88.2822265625, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=6.596158480564634, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7600034834708445
   [junit4]   2> 502498 T2148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1235ce5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@93791f),segFN=segments_1,generation=1}
   [junit4]   2> 502498 T2148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502499 T2148 oass.SolrIndexSearcher.<init> Opening Searcher@180be7e main
   [junit4]   2> 502499 T2161 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 502500 T2161 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 502500 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 502501 T2161 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502501 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502502 T2161 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 502502 T2161 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502503 T2200 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@180be7e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 502503 T2187 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 502503 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502504 T2161 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 502505 T2161 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 502506 T2161 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 502506 T2161 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 502506 T2148 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 502506 T2187 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 502506 T2161 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 502506 T2148 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:55739 collection:solrj_collection shard:shard1
   [junit4]   2> 502508 T2161 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 502508 T2161 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 502509 T2161 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 502509 T2148 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 502510 T2161 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 502512 T2161 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 502516 T2161 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 502516 T2161 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 502516 T2148 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 502517 T2161 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=88.2822265625, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=6.596158480564634, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7600034834708445
   [junit4]   2> 502517 T2161 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1d740c6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@72c648),segFN=segments_1,generation=1}
   [junit4]   2> 502518 T2161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502518 T2161 oass.SolrIndexSearcher.<init> Opening Searcher@3e7eb1 main
   [junit4]   2> 502518 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502519 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502519 T2148 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 502519 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502521 T2135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 502522 T2187 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502522 T2202 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@3e7eb1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 502524 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502525 T2161 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
   [junit4]   2> 502525 T2161 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:41810 collection:solrj_collection shard:shard2
   [junit4]   2> 502526 T2139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502526 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502526 T2187 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502526 T2166 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502526 T2152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502527 T2161 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 502526 T2180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502526 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 502532 T2187 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502533 T2187 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502533 T2161 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 502534 T2187 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502534 T2187 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502535 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502536 T2187 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502536 T2161 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=179999
   [junit4]   2> 502536 T2187 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502537 T2135 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 502537 T2187 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 502538 T2187 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/data/
   [junit4]   2> 502538 T2187 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@134f867
   [junit4]   2> 502539 T2187 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/data
   [junit4]   2> 502540 T2187 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 502540 T2134 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 502540 T2187 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 502541 T2187 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1389961088611/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 502541 T2187 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5037992432018531]
   [junit4]   2> 502542 T2187 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@e04232 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5abba8),segFN=segments_1,generation=1}
   [junit4]   2> 502542 T2187 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502545 T2187 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 502546 T2187 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 502546 T2187 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 502546 T2187 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502547 T2187 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502547 T2187 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 502547 T2187 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502548 T2187 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 502548 T2187 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 502549 T2187 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 502549 T2187 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 502549 T2187 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 502550 T2187 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 502550 T2187 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 502551 T2187 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 502551 T2187 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 502552 T2187 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 502553 T2187 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 502556 T2187 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 502557 T2187 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 502557 T2187 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=88.2822265625, floorSegmentMB=2.1708984375, forceMergeDeletesPctAllowed=6.596158480564634, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7600034834708445
   [junit4]   2> 502558 T2187 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@e04232 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5abba8),segFN=segments_1,generation=1}
   [junit4]   2> 502558 T2187 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502558 T2187 oass.SolrIndexSearcher.<init> Opening Searcher@1bebd5 main
   [junit4]   2> 502561 T2204 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@1bebd5 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 502563 T2187 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 502563 T2187 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:37677 collection:solrj_collection shard:shard2
   [junit4]   2> 502566 T2176 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 502575 T2176 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 502580 T2176 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 502593 T2176 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502598 T2176 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 502603 T2176 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502604 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502604 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502605 T2176 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 502606 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 502606 T2176 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 502607 T2176 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 502607 T2176 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/data/
   [junit4]   2> 502608 T2176 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@134f867
   [junit4]   2> 502609 T2176 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/data
   [junit4]   2> 502609 T2176 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 502610 T2176 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 502610 T2176 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1389961087408/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 502611 T2176 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5037992432018531]
   [junit4]   2> 502612 T2176 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@143a9ad lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@a6d164),segFN=segments_1,generation=1}
   [junit4]   2> 502612 T2176 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 502617 T2176 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 502618 T2176 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 502618 T2176 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 502618 T2176 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 502619 T2176 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdatePro

[...truncated too long message...]

 [junit4]   2> 1150344 T5022 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 49680 T4922 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 69773 T4922 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest: 
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 69774 T4922 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]   2> 72776 T4922 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {_version_=PostingsFormat(name=FSTPulsing41), n_ti=Lucene41(blocksize=128), id=MockFixedIntBlock(blockSize=1765)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=lt, timezone=Antarctica/Casey
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=2,free=213963288,total=314310656
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsCollectionsAPIDistributedZkTest, BasicDistributedZk2Test, TestStressUserVersions, TestUpdate, UniqFieldsUpdateProcessorFactoryTest, TestSchemaVersionResource, TestQuerySenderNoQuery, SimplePostToolTest, AliasIntegrationTest, CurrencyFieldOpenExchangeTest, TestSolr4Spatial, ConnectionManagerTest, TestStressLucene, PluginInfoTest, TestIndexingPerformance, TimeZoneUtilsTest, TestFuzzyAnalyzedSuggestions, TestIBSimilarityFactory, AddSchemaFieldsUpdateProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, BlockDirectoryTest, TestAddFieldRealTimeGet, TestReload, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, TestFiltering, ShardRoutingCustomTest, TestLazyCores, PreAnalyzedUpdateProcessorTest, TestFoldingMultitermQuery, JSONWriterTest, TestPartialUpdateDeduplication, TestRandomMergePolicy, QueryFacetTest, PingRequestHandlerTest, SpellCheckComponentTest, BasicZkTest, DirectSolrSpellCheckerTest, SolrCmdDistributorTest, SuggesterWFSTTest, TestCSVResponseWriter, TestManagedSchemaFieldResource, DistributedQueryComponentCustomSortTest, SolrIndexConfigTest, TestDistribDocBasedVersion, TestRandomFaceting, TestFieldResource, TestNumberUtils, TestUtils, TestJmxIntegration, TestMaxScoreQueryParser, AddBlockUpdateTest, TestSolrDeletionPolicy2, TestSolrXmlPersistor, TestSerializedLuceneMatchVersion, TestCSVLoader, SolrRequestParserTest, DefaultValueUpdateProcessorTest, SignatureUpdateProcessorFactoryTest, TestBinaryField, TestFieldCollectionResource, ConvertedLegacyTest, PrimUtilsTest, NoCacheHeaderTest, TestXIncludeConfig, StressHdfsTest, CoreContainerCoreInitFailuresTest, CachingDirectoryFactoryTest, TestCoreDiscovery, ChaosMonkeySafeLeaderTest, TestMultiCoreConfBootstrap, OpenExchangeRatesOrgProviderTest, TestSolrCoreProperties, TestOmitPositions, TestCursorMarkWithoutUniqueKey, TestDefaultSimilarityFactory, TestClassNameShortening, TestSolrXmlPersistence, TestWordDelimiterFilterFactory, SpellCheckCollatorTest, NumericFieldsTest, XmlUpdateRequestHandlerTest, HdfsSyncSliceTest, TestImplicitCoreProperties, FieldFacetExtrasTest, TestSweetSpotSimilarityFactory, TestUniqueKeyFieldResource, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, ClusterStateUpdateTest, LeaderElectionTest, TestZkChroot, TestDistributedSearch, TestDistributedGrouping, TestHashPartitioner, TestStressReorder, HardAutoCommitTest, TestRangeQuery, SimpleFacetsTest, StatsComponentTest, TestGroupingSearch, TestSort, DirectUpdateHandlerTest, TestIndexSearcher, HighlighterTest, TestExtendedDismaxParser, SpatialFilterTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, QueryEqualityTest, TestRemoteStreaming, DocValuesMultiTest, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, TestQueryTypes, RequestHandlersTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestBinaryResponseWriter, TestQuerySenderListener, TestSolrIndexConfig, BadComponentTest, MultiTermTest, SampleTest, MinimalSchemaTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestCodecSupport, EchoParamsTest, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TestFastOutputStream, URLClassifyProcessorTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, SolrTestCaseJ4Test, TestDistributedMissingSort, AbstractAnalyticsStatsTest, NoFacetTest, FieldFacetTest, RangeFacetTest, AssignTest, DeleteReplicaTest, DeleteShardTest, MigrateRouteKeyTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MigrateRouteKeyTest -Dtests.seed=C42072D286CD121B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt -Dtests.timezone=Antarctica/Casey -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | MigrateRouteKeyTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.MigrateRouteKeyTest: 
   [junit4]    >    1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C42072D286CD121B]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=5059, name=Timer-33, state=WAITING, group=TGRP-MigrateRouteKeyTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C42072D286CD121B]:0)
   [junit4] Completed on J1 in 72.83s, 1 test, 2 errors <<< FAILURES!

[...truncated 145 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 364 suites, 1596 tests, 2 suite-level errors, 1 error, 134 ignored (6 assumptions)

Total time: 46 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b02) - Build # 9129 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9129/
Java: 32bit/jdk1.7.0_60-ea-b02 -client -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
could not set the new overseer

Stack Trace:
java.lang.AssertionError: could not set the new overseer
	at __randomizedtesting.SeedInfo.seed([7342B65A29AD0150:F2A438425EF2616C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.OverseerRolesTest.addOverseerRole2ExistingNodes(OverseerRolesTest.java:120)
	at org.apache.solr.cloud.OverseerRolesTest.doTest(OverseerRolesTest.java:86)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10276 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 637125 T2319 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /qz/pi
   [junit4]   2> 637129 T2319 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1389973481734
   [junit4]   2> 637129 T2319 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 637130 T2320 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 637230 T2319 oasc.ZkTestServer.run start zk server on port:34662
   [junit4]   2> 637231 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637303 T2326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10e61b6 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637303 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637304 T2319 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 637308 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637309 T2328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d5011 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637309 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637310 T2319 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 637313 T2319 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 637316 T2319 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 637318 T2319 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 637320 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 637321 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 637324 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 637324 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 637327 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 637327 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 637329 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 637330 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 637332 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 637333 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 637334 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 637335 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 637337 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 637337 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 637339 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 637340 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 637342 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 637342 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 637345 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 637345 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 637347 T2319 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 637348 T2319 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 637438 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 637441 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33250
   [junit4]   2> 637442 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 637442 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 637442 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955
   [junit4]   2> 637443 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/'
   [junit4]   2> 637454 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/solr.xml
   [junit4]   2> 637481 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 637494 T2319 oasc.CoreContainer.<init> New CoreContainer 951641
   [junit4]   2> 637494 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/]
   [junit4]   2> 637496 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 637496 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 637496 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 637497 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 637497 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 637497 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 637497 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 637498 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 637498 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 637500 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 637501 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 637501 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 637501 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 637502 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 637502 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637504 T2339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154cd7b name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637504 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637506 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637507 T2341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12a4a5a name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637507 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637509 T2319 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 637511 T2319 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 637513 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 637515 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33250_qz%2Fpi
   [junit4]   2> 637515 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33250_qz%2Fpi
   [junit4]   2> 637517 T2319 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 637519 T2319 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 637522 T2319 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 637524 T2319 oasc.Overseer.start Overseer (id=91093302099116035-127.0.0.1:33250_qz%2Fpi-n_0000000000) starting
   [junit4]   2> 637526 T2319 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 637529 T2343 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 637529 T2319 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 637529 T2343 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 637531 T2319 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 637533 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 637536 T2342 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 637537 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955
   [junit4]   2> 637538 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/conf/
   [junit4]   2> 637570 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 637570 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 637571 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 637572 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637573 T2346 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9941d3 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637573 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637574 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 637576 T2319 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 637642 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 637643 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36749
   [junit4]   2> 637644 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 637645 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 637645 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181
   [junit4]   2> 637645 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/'
   [junit4]   2> 637657 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/solr.xml
   [junit4]   2> 637692 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 637692 T2319 oasc.CoreContainer.<init> New CoreContainer 22109912
   [junit4]   2> 637692 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/]
   [junit4]   2> 637693 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 637694 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 637694 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 637694 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 637694 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 637695 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 637695 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 637695 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 637696 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 637698 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 637698 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 637699 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 637699 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 637699 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 637700 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637701 T2357 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129d240 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637702 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637703 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637704 T2359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5726b name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637704 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637708 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 638711 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36749_qz%2Fpi
   [junit4]   2> 638712 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36749_qz%2Fpi
   [junit4]   2> 638714 T2341 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 638715 T2359 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 638715 T2346 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 638719 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181
   [junit4]   2> 638721 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/conf/
   [junit4]   2> 638754 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 638754 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 638755 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 638824 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 638825 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40327
   [junit4]   2> 638826 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 638826 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 638827 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360
   [junit4]   2> 638827 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/'
   [junit4]   2> 638839 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/solr.xml
   [junit4]   2> 638874 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 638875 T2319 oasc.CoreContainer.<init> New CoreContainer 15544662
   [junit4]   2> 638875 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/]
   [junit4]   2> 638876 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 638876 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 638876 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 638877 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 638877 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 638877 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 638878 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 638878 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 638878 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 638880 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 638881 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 638881 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 638881 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 638882 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 638882 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638884 T2371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@424634 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638884 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638886 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 638887 T2373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fd35d1 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 638887 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 638890 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 639893 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40327_qz%2Fpi
   [junit4]   2> 639894 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40327_qz%2Fpi
   [junit4]   2> 639897 T2359 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 639898 T2346 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 639897 T2373 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 639897 T2341 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 639903 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360
   [junit4]   2> 639904 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/conf/
   [junit4]   2> 639935 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 639936 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 639936 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 640005 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 640007 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52992
   [junit4]   2> 640008 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 640008 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 640009 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542
   [junit4]   2> 640009 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/'
   [junit4]   2> 640021 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/solr.xml
   [junit4]   2> 640048 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 640048 T2319 oasc.CoreContainer.<init> New CoreContainer 10662706
   [junit4]   2> 640048 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/]
   [junit4]   2> 640049 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 640050 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 640050 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 640050 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 640051 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 640051 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 640051 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 640051 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 640052 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 640054 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 640054 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 640055 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 640055 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 640055 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 640056 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 640058 T2385 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a2ac75 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 640058 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 640060 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 640061 T2387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18b985d name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 640061 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 640066 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 641068 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52992_qz%2Fpi
   [junit4]   2> 641070 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52992_qz%2Fpi
   [junit4]   2> 641076 T2359 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 641076 T2346 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 641076 T2341 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 641077 T2373 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 641077 T2387 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 641081 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542
   [junit4]   2> 641082 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/conf/
   [junit4]   2> 641114 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 641115 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 641115 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 641201 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 641203 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39805
   [junit4]   2> 641204 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 641204 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 641205 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720
   [junit4]   2> 641205 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/'
   [junit4]   2> 641217 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/solr.xml
   [junit4]   2> 641243 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 641244 T2319 oasc.CoreContainer.<init> New CoreContainer 25015211
   [junit4]   2> 641244 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/]
   [junit4]   2> 641245 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 641245 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 641246 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 641246 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 641246 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 641247 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 641247 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 641247 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 641247 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 641250 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 641250 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 641250 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 641251 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 641251 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 641252 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 641253 T2399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19ea571 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 641253 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 641255 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 641256 T2401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101b459 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 641256 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 641259 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 642261 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39805_qz%2Fpi
   [junit4]   2> 642262 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39805_qz%2Fpi
   [junit4]   2> 642265 T2373 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642265 T2346 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642266 T2341 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642265 T2401 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642265 T2359 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642266 T2387 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 642271 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720
   [junit4]   2> 642273 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/conf/
   [junit4]   2> 642317 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 642318 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 642318 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 642390 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 642392 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58265
   [junit4]   2> 642393 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 642393 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 642394 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924
   [junit4]   2> 642394 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/'
   [junit4]   2> 642406 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/solr.xml
   [junit4]   2> 642436 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 642437 T2319 oasc.CoreContainer.<init> New CoreContainer 29212059
   [junit4]   2> 642437 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/]
   [junit4]   2> 642438 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 642439 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 642439 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 642440 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 642440 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 642441 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 642441 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 642442 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 642442 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 642451 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 642452 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 642452 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 642452 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 642453 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 642453 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 642455 T2413 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56a8d8 name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 642455 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 642456 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 642458 T2415 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@174e3a9 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 642458 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 642461 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 643463 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58265_qz%2Fpi
   [junit4]   2> 643466 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58265_qz%2Fpi
   [junit4]   2> 643468 T2346 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643468 T2341 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643468 T2415 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643469 T2401 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643469 T2373 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643469 T2359 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643469 T2387 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 643473 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924
   [junit4]   2> 643474 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/conf/
   [junit4]   2> 643501 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 643501 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 643502 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 643604 T2319 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 643606 T2319 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49585
   [junit4]   2> 643607 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 643608 T2319 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 643608 T2319 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108
   [junit4]   2> 643609 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/'
   [junit4]   2> 643620 T2319 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/solr.xml
   [junit4]   2> 643647 T2319 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 643648 T2319 oasc.CoreContainer.<init> New CoreContainer 18737671
   [junit4]   2> 643648 T2319 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/]
   [junit4]   2> 643649 T2319 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 643649 T2319 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 643650 T2319 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 643650 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 643650 T2319 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 643650 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 643651 T2319 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 643651 T2319 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 643651 T2319 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 643653 T2319 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 643654 T2319 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 643654 T2319 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 643655 T2319 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34662/solr
   [junit4]   2> 643655 T2319 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 643655 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 643657 T2427 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@efa56e name:ZooKeeperConnection Watcher:127.0.0.1:34662 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 643657 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 643659 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 643660 T2429 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20be32 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 643660 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 643664 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 644668 T2319 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49585_qz%2Fpi
   [junit4]   2> 644669 T2319 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49585_qz%2Fpi
   [junit4]   2> 644671 T2401 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2415 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2387 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2373 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2341 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2359 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2346 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644672 T2429 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 644677 T2319 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108
   [junit4]   2> 644678 T2319 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/conf/
   [junit4]   2> 644714 T2319 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 644715 T2319 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 644715 T2319 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 644716 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 644719 T2432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1574c5 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 644719 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 644720 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 644723 T2319 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 644724 T2434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d210e6 name:ZooKeeperConnection Watcher:127.0.0.1:34662/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 644724 T2319 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 644725 T2319 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 644728 T2424 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&version=2&maxShardsPerNode=2&wt=javabin&action=CREATE&replicationFactor=2&name=testOverseerCol
   [junit4]   2> 644729 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644730 T2343 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 644731 T2343 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 644731 T2343 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 644731 T2343 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 644732 T2343 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 644735 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644737 T2342 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 644738 T2342 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 644738 T2342 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 644740 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644741 T2373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2346 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2434 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644741 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644741 T2429 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644742 T2415 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644744 T2432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644836 T2343 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 644836 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:36749_qz%2Fpi
   [junit4]   2> 644836 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:33250_qz%2Fpi
   [junit4]   2> 644838 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:40327_qz%2Fpi
   [junit4]   2> 644838 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:49585_qz%2Fpi
   [junit4]   2> 644839 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:58265_qz%2Fpi
   [junit4]   2> 644839 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:39805_qz%2Fpi
   [junit4]   2> 644839 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:52992_qz%2Fpi
   [junit4]   2> 644840 T2352 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 644840 T2368 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 644840 T2343 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:36749_qz%2Fpi
   [junit4]   2> 644840 T2334 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 644842 T2397 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 644843 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644843 T2408 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 644842 T2378 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 644844 T2354 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 644842 T2352 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644842 T2423 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 644846 T2368 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644847 T2397 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644847 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644847 T2378 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644848 T2408 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644847 T2423 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644847 T2354 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644847 T2334 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 644848 T2342 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 644850 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36749/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36749_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644853 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40327/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40327_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644855 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644856 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33250/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33250_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644858 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644859 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39805/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39805_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644861 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644862 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52992/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52992_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644865 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644866 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58265/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58265_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644868 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644869 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36749/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36749_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644871 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644872 T2342 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49585/qz/pi",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49585_qz%2Fpi",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 644875 T2341 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 644978 T2373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2415 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644979 T2432 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2429 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2346 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644978 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644979 T2434 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 644979 T2359 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 645845 T2352 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 645846 T2352 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard1_replica1
   [junit4]   2> 645846 T2368 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 645846 T2352 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645847 T2368 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/testOverseerCol_shard2_replica1
   [junit4]   2> 645847 T2368 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645847 T2397 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 645848 T2408 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 645849 T2423 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 645849 T2408 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1
   [junit4]   2> 645849 T2408 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645850 T2423 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/testOverseerCol_shard2_replica2
   [junit4]   2> 645850 T2423 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645848 T2378 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 645851 T2423 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645851 T2423 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645848 T2352 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645852 T2352 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645851 T2378 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/testOverseerCol_shard4_replica1
   [junit4]   2> 645853 T2378 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645850 T2408 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645850 T2368 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645854 T2368 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645849 T2334 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 645849 T2354 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 645849 T2397 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2
   [junit4]   2> 645855 T2397 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645855 T2334 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/testOverseerCol_shard1_replica2
   [junit4]   2> 645856 T2334 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645854 T2378 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645853 T2408 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645853 T2352 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard1_replica1/'
   [junit4]   2> 645852 T2423 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389973488108/testOverseerCol_shard2_replica2/'
   [junit4]   2> 645857 T2368 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1389973483360/testOverseerCol_shard2_replica1/'
   [junit4]   2> 645857 T2334 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645858 T2408 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/'
   [junit4]   2> 645856 T2378 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645856 T2397 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645855 T2354 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2
   [junit4]   2> 645859 T2397 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645858 T2334 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645860 T2354 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 645860 T2378 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389973484542/testOverseerCol_shard4_replica1/'
   [junit4]   2> 645862 T2354 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 645862 T2354 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 645863 T2397 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/'
   [junit4]   2> 645863 T2334 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389973481955/testOverseerCol_shard1_replica2/'
   [junit4]   2> 645868 T2354 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/'
   [junit4]   2> 645955 T2397 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645964 T2334 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645965 T2408 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 645967 T2352 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 646002 T2423 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 646008 T2354 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 646012 T2368 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 646021 T2378 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 646058 T2408 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646060 T2408 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646067 T2397 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646094 T2397 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646096 T2408 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 646098 T2368 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646098 T2334 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646099 T2368 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646101 T2352 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646103 T2352 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646103 T2334 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646110 T2397 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 646112 T2354 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646114 T2368 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 646114 T2354 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646114 T2423 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646116 T2423 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646129 T2352 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 646130 T2423 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 646153 T2334 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 646164 T2354 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 646172 T2378 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 646174 T2378 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 646208 T2378 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 646948 T2354 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 646978 T2397 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 647009 T2397 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647010 T2354 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647011 T2397 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647015 T2354 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647039 T2423 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 647044 T2397 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647048 T2354 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647049 T2397 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647052 T2408 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 647054 T2397 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 647055 T2397 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 647056 T2397 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 647056 T2397 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 647058 T2397 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 647058 T2397 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 647059 T2397 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 647059 T2397 oasc.SolrCore.<init> [testOverseerCol_shard3_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/data/
   [junit4]   2> 647060 T2397 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ebb27a
   [junit4]   2> 647060 T2408 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647062 T2354 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647064 T2408 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647071 T2397 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/data
   [junit4]   2> 647072 T2397 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/data/index/
   [junit4]   2> 647072 T2397 oasc.SolrCore.initIndex WARN [testOverseerCol_shard3_replica2] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 647076 T2334 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 647075 T2423 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647074 T2354 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 647077 T2397 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389973485720/testOverseerCol_shard3_replica2/data/index
   [junit4]   2> 647080 T2397 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=0.3525390625, floorSegmentMB=0.9072265625, forceMergeDeletesPctAllowed=16.100790459691478, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.19014597527715332
   [junit4]   2> 647082 T2397 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ec87b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb773d),segFN=segments_1,generation=1}
   [junit4]   2> 647082 T2397 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 647083 T2368 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 647081 T2354 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 647085 T2354 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 647085 T2354 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 647086 T2334 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647087 T2354 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 647087 T2354 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 647088 T2354 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 647088 T2354 oasc.SolrCore.<init> [testOverseerCol_shard4_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/data/
   [junit4]   2> 647089 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647088 T2397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 647090 T2334 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647091 T2368 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 647089 T2354 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ebb27a
   [junit4]   2> 647092 T2423 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647094 T2354 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/data
   [junit4]   2> 647094 T2354 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/data/index/
   [junit4]   2> 647090 T2397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 647095 T2397 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 647096 T2397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 647096 T2397 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 647095 T2354 oasc.SolrCore.initIndex WARN [testOverseerCol_shard4_replica2] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 647094 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 647097 T2354 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389973482181/testOverseerCol_shard4_replica2/data/index
   [junit4]   2> 647097 T2397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 647096 T2368 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 647099 T2397 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 647098 T2354 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=0.3525390625, floorSegmentM

[...truncated too long message...]

ref count has reached 0 - closing IndexWriter
   [junit4]   2> 665330 T2415 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 665331 T2415 oasc.SolrCore.closeSearcher [testOverseerCol_shard3_replica1] Closing main searcher on request.
   [junit4]   2> 665331 T2415 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 665331 T2415 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data/index;done=false>>]
   [junit4]   2> 665331 T2415 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data/index
   [junit4]   2> 665331 T2415 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data;done=false>>]
   [junit4]   2> 665332 T2415 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389973486924/testOverseerCol_shard3_replica1/data
   [junit4]   2> 665332 T2415 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leaders/shard3
   [junit4]   2> 665332 T2415 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 665333 T2415 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard3/election/91093302099116046-core_node6-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 665333 T2415 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 665333 T2415 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=sr_ME, timezone=Africa/Windhoek
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=144988144,total=231301120
   [junit4]   2> NOTE: All tests run in this JVM: [TermVectorComponentTest, TestCollapseQParserPlugin, SolrIndexSplitterTest, ShardSplitTest, TestBM25SimilarityFactory, SpellPossibilityIteratorTest, TestReloadAndDeleteDocs, ShardRoutingTest, TestCloudManagedSchemaAddField, IndexSchemaTest, TestModifyConfFiles, RegexBoostProcessorTest, PeerSyncTest, JsonLoaderTest, TestSearchPerf, FunctionTest, RangeFacetTest, LukeRequestHandlerTest, TestSolrQueryParserResource, DOMUtilTest, TestReversedWildcardFilterFactory, ClusterStateUpdateTest, WordBreakSolrSpellCheckerTest, OverseerTest, TestAnalyzeInfixSuggestions, TestCollationFieldDocValues, TestConfig, AnalysisAfterCoreReloadTest, DeleteReplicaTest, HdfsRecoveryZkTest, InfoHandlerTest, AutoCommitTest, ScriptEngineTest, TestJoin, SchemaVersionSpecificBehaviorTest, TestFaceting, TestDynamicFieldResource, SolrInfoMBeanTest, ExternalFileFieldSortTest, TestDocSet, TestMergePolicyConfig, SuggesterFSTTest, SearchHandlerTest, TestPseudoReturnFields, ParsingFieldUpdateProcessorsTest, TestGroupingSearch, TestInfoStreamLogging, TestSimpleQParserPlugin, TestManagedSchema, QueryEqualityTest, UUIDFieldTest, DistributedSpellCheckComponentTest, CoreMergeIndexesAdminHandlerTest, OverseerCollectionProcessorTest, TestSystemIdResolver, TestWriterPerf, ExpressionTest, NoFacetTest, TestCoreContainer, TestFieldTypeCollectionResource, CoreAdminCreateDiscoverTest, CSVRequestHandlerTest, TestTrie, TestLFUCache, RecoveryZkTest, HdfsBasicDistributedZk2Test, QueryParsingTest, TestSolrXml, OverseerRolesTest]
   [junit4] Completed on J0 in 29.18s, 1 test, 1 failure <<< FAILURES!

[...truncated 641 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 364 suites, 1596 tests, 1 failure, 34 ignored (6 assumptions)

Total time: 51 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b02 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9128 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9128/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
could not set the new overseer

Stack Trace:
java.lang.AssertionError: could not set the new overseer
	at __randomizedtesting.SeedInfo.seed([D50F977CCEAE294B:54E91964B9F14977]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.OverseerRolesTest.addOverseerRole2ExistingNodes(OverseerRolesTest.java:120)
	at org.apache.solr.cloud.OverseerRolesTest.doTest(OverseerRolesTest.java:86)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10786 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 1048102 T3812 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1048105 T3812 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1389967405430
   [junit4]   2> 1048106 T3812 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1048106 T3813 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1048206 T3812 oasc.ZkTestServer.run start zk server on port:60040
   [junit4]   2> 1048207 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048234 T3819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2008a8 name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048234 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048235 T3812 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1048238 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048239 T3821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78f9df name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048239 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048239 T3812 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1048241 T3812 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1048243 T3812 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1048244 T3812 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1048245 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1048246 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1048248 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1048248 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1048250 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1048251 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1048252 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1048252 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1048255 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1048255 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1048257 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1048257 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1048259 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1048259 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1048261 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1048262 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1048263 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1048264 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1048265 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1048266 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1048268 T3812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1048268 T3812 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1048357 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1048359 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38041
   [junit4]   2> 1048359 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1048359 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1048360 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594
   [junit4]   2> 1048360 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/'
   [junit4]   2> 1048368 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/solr.xml
   [junit4]   2> 1048398 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1048399 T3812 oasc.CoreContainer.<init> New CoreContainer 28556287
   [junit4]   2> 1048399 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/]
   [junit4]   2> 1048400 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1048400 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1048401 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1048401 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1048401 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1048401 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1048402 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1048402 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1048402 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1048404 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1048404 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1048405 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1048405 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1048405 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1048406 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048407 T3832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fb77bc name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048408 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048409 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048410 T3834 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef01a9 name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048410 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048411 T3812 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1048413 T3812 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1048415 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1048416 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38041_
   [junit4]   2> 1048417 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38041_
   [junit4]   2> 1048418 T3812 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1048419 T3812 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1048422 T3812 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1048424 T3812 oasc.Overseer.start Overseer (id=91092903882391555-127.0.0.1:38041_-n_0000000000) starting
   [junit4]   2> 1048428 T3812 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1048430 T3836 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1048430 T3812 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1048430 T3836 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1048432 T3812 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1048433 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1048434 T3835 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1048436 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594
   [junit4]   2> 1048437 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/conf/
   [junit4]   2> 1048458 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1048458 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1048458 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1048459 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048460 T3839 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17ae19c name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048460 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048461 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1048463 T3812 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1048524 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1048526 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37847
   [junit4]   2> 1048526 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1048527 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1048527 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787
   [junit4]   2> 1048527 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/'
   [junit4]   2> 1048539 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/solr.xml
   [junit4]   2> 1048570 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1048571 T3812 oasc.CoreContainer.<init> New CoreContainer 32580417
   [junit4]   2> 1048571 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/]
   [junit4]   2> 1048572 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1048572 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1048573 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1048573 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1048573 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1048573 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1048574 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1048574 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1048574 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1048576 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1048576 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1048577 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1048577 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1048577 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1048578 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048579 T3850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4196f0 name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048579 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048585 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048585 T3852 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a8b5 name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048586 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048588 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1049590 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37847_
   [junit4]   2> 1049591 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37847_
   [junit4]   2> 1049593 T3839 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049593 T3852 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049593 T3834 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049596 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787
   [junit4]   2> 1049597 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/conf/
   [junit4]   2> 1049618 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1049618 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1049618 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1049679 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1049680 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46506
   [junit4]   2> 1049681 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1049681 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1049681 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943
   [junit4]   2> 1049682 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/'
   [junit4]   2> 1049690 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/solr.xml
   [junit4]   2> 1049720 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1049721 T3812 oasc.CoreContainer.<init> New CoreContainer 1987910
   [junit4]   2> 1049721 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/]
   [junit4]   2> 1049722 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1049722 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1049722 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1049723 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1049723 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1049723 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1049723 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1049724 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1049724 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1049726 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1049726 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1049726 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1049727 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1049727 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1049728 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1049729 T3864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1708ba7 name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1049729 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1049730 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1049731 T3866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@790d4 name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1049732 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1049737 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1050739 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46506_
   [junit4]   2> 1050740 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46506_
   [junit4]   2> 1050742 T3839 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1050742 T3866 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1050742 T3852 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1050742 T3834 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1050745 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943
   [junit4]   2> 1050746 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/conf/
   [junit4]   2> 1050774 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1050774 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1050775 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1050836 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1050838 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39219
   [junit4]   2> 1050838 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1050839 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1050839 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099
   [junit4]   2> 1050839 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/'
   [junit4]   2> 1050853 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/solr.xml
   [junit4]   2> 1050902 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1050903 T3812 oasc.CoreContainer.<init> New CoreContainer 15641993
   [junit4]   2> 1050903 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/]
   [junit4]   2> 1050904 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1050904 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1050904 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1050904 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1050905 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1050905 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1050905 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1050906 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1050906 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1050908 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1050908 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1050908 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1050909 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1050909 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1050910 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1050911 T3878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c328ef name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1050911 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1050913 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1050914 T3880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1104aed name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1050914 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1050917 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1051919 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39219_
   [junit4]   2> 1051921 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39219_
   [junit4]   2> 1051923 T3880 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1051923 T3866 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1051923 T3852 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1051923 T3839 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1051923 T3834 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1051928 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099
   [junit4]   2> 1051929 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/conf/
   [junit4]   2> 1051959 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1051959 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1051959 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1052020 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1052022 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60081
   [junit4]   2> 1052022 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1052023 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1052023 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284
   [junit4]   2> 1052023 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/'
   [junit4]   2> 1052032 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/solr.xml
   [junit4]   2> 1052062 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1052062 T3812 oasc.CoreContainer.<init> New CoreContainer 21730220
   [junit4]   2> 1052062 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/]
   [junit4]   2> 1052063 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1052064 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1052064 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1052064 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1052064 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1052065 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1052065 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1052065 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1052065 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1052067 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1052068 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1052068 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1052068 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1052069 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1052070 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1052071 T3892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc3b50 name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1052071 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1052073 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1052074 T3894 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e408b name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1052075 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1052077 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1053079 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60081_
   [junit4]   2> 1053080 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60081_
   [junit4]   2> 1053082 T3839 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053082 T3834 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053082 T3866 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053082 T3852 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053083 T3894 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053082 T3880 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1053086 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284
   [junit4]   2> 1053087 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/conf/
   [junit4]   2> 1053116 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1053116 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1053117 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1053182 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1053184 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34949
   [junit4]   2> 1053184 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1053185 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1053185 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441
   [junit4]   2> 1053185 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/'
   [junit4]   2> 1053194 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/solr.xml
   [junit4]   2> 1053226 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1053226 T3812 oasc.CoreContainer.<init> New CoreContainer 17032387
   [junit4]   2> 1053226 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/]
   [junit4]   2> 1053227 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1053228 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1053228 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1053228 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1053228 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1053229 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1053229 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1053229 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1053229 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1053231 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1053232 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1053232 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1053232 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1053232 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1053233 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1053235 T3906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fff5a name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1053235 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1053237 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1053238 T3908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@659a06 name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1053238 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1053240 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1054243 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34949_
   [junit4]   2> 1054244 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34949_
   [junit4]   2> 1054246 T3866 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3834 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3894 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3852 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3908 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3839 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054246 T3880 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1054250 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441
   [junit4]   2> 1054251 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/conf/
   [junit4]   2> 1054279 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1054280 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1054280 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1054342 T3812 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1054343 T3812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41281
   [junit4]   2> 1054344 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1054344 T3812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1054344 T3812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604
   [junit4]   2> 1054344 T3812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/'
   [junit4]   2> 1054353 T3812 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/solr.xml
   [junit4]   2> 1054385 T3812 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1054385 T3812 oasc.CoreContainer.<init> New CoreContainer 11367880
   [junit4]   2> 1054386 T3812 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/]
   [junit4]   2> 1054387 T3812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1054387 T3812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1054387 T3812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1054387 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1054388 T3812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1054388 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1054388 T3812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1054388 T3812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1054389 T3812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1054390 T3812 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1054391 T3812 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1054391 T3812 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1054391 T3812 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60040/solr
   [junit4]   2> 1054392 T3812 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1054392 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1054393 T3920 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fcbfa name:ZooKeeperConnection Watcher:127.0.0.1:60040 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1054394 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1054395 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1054396 T3922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@765d43 name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1054396 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1054398 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055400 T3812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41281_
   [junit4]   2> 1055401 T3812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41281_
   [junit4]   2> 1055403 T3839 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3866 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3908 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3922 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3894 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3852 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055404 T3880 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055403 T3834 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1055408 T3812 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604
   [junit4]   2> 1055409 T3812 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/conf/
   [junit4]   2> 1055434 T3812 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1055435 T3812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1055435 T3812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1055436 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055438 T3925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d3c93e name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055438 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1055439 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055441 T3812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055442 T3927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c496c name:ZooKeeperConnection Watcher:127.0.0.1:60040/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055442 T3812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1055443 T3812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055446 T3827 oasha.CollectionsHandler.handleCreateAction Creating Collection : maxShardsPerNode=2&replicationFactor=2&name=testOverseerCol&wt=javabin&version=2&numShards=4&action=CREATE
   [junit4]   2> 1055448 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055449 T3836 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1055450 T3836 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1055450 T3836 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1055450 T3836 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 1055450 T3836 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 1055453 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055453 T3835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1055454 T3835 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 1055454 T3835 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 1055455 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055456 T3839 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3866 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3927 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055456 T3922 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055553 T3836 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 1055553 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:46506_
   [junit4]   2> 1055553 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:37847_
   [junit4]   2> 1055554 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:34949_
   [junit4]   2> 1055557 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:41281_
   [junit4]   2> 1055558 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:39219_
   [junit4]   2> 1055558 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:60081_
   [junit4]   2> 1055558 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:38041_
   [junit4]   2> 1055561 T3836 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:46506_
   [junit4]   2> 1055562 T3847 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 1055563 T3903 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 1055563 T3861 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 1055564 T3847 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055564 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055570 T3903 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055564 T3828 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 1055571 T3874 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 1055570 T3918 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 1055570 T3861 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055572 T3828 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055570 T3888 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 1055572 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055573 T3835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1055571 T3860 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 1055573 T3888 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055573 T3874 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055574 T3860 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055574 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37847",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37847_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055573 T3918 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055579 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34949",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34949_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055581 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055583 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46506",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46506_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055585 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055585 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38041",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38041_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055587 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055588 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39219",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39219_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055589 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055590 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41281",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41281_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055591 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055593 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60081",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60081_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055597 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055598 T3835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46506",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46506_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055600 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055701 T3852 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3922 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3927 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3839 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1055701 T3866 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1056570 T3847 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 1056571 T3903 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 1056571 T3847 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/testOverseerCol_shard1_replica2
   [junit4]   2> 1056571 T3847 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056571 T3903 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1
   [junit4]   2> 1056572 T3903 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056572 T3861 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 1056573 T3903 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056573 T3903 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056572 T3828 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 1056572 T3847 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056574 T3847 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056574 T3888 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 1056573 T3861 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/testOverseerCol_shard1_replica1
   [junit4]   2> 1056574 T3847 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1389967405787/testOverseerCol_shard1_replica2/'
   [junit4]   2> 1056575 T3888 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/testOverseerCol_shard3_replica2
   [junit4]   2> 1056575 T3888 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056574 T3860 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 1056574 T3874 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 1056574 T3903 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/'
   [junit4]   2> 1056576 T3874 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/testOverseerCol_shard3_replica1
   [junit4]   2> 1056576 T3874 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056574 T3828 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/testOverseerCol_shard4_replica1
   [junit4]   2> 1056577 T3828 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056576 T3860 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/testOverseerCol_shard4_replica2
   [junit4]   2> 1056577 T3860 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056575 T3918 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 1056574 T3861 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056577 T3888 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056579 T3888 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056577 T3874 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056579 T3874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056579 T3918 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2
   [junit4]   2> 1056580 T3918 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1056580 T3860 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056580 T3860 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056580 T3828 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056581 T3828 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056581 T3860 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/testOverseerCol_shard4_replica2/'
   [junit4]   2> 1056580 T3861 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056582 T3828 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1389967405594/testOverseerCol_shard4_replica1/'
   [junit4]   2> 1056582 T3861 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056581 T3918 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056583 T3918 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1056580 T3888 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1389967409284/testOverseerCol_shard3_replica2/'
   [junit4]   2> 1056580 T3874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1389967408099/testOverseerCol_shard3_replica1/'
   [junit4]   2> 1056583 T3861 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1389967406943/testOverseerCol_shard1_replica1/'
   [junit4]   2> 1056585 T3918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/'
   [junit4]   2> 1056639 T3847 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056651 T3860 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056656 T3918 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056664 T3888 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056669 T3903 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056671 T3861 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056700 T3874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056727 T3828 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056739 T3918 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056740 T3918 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056751 T3888 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056751 T3847 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056753 T3888 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056755 T3918 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 1056756 T3847 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056765 T3888 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 1056773 T3847 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 1056779 T3861 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056780 T3861 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056794 T3860 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056795 T3860 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056809 T3861 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 1056812 T3860 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 1056818 T3874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056823 T3874 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056828 T3903 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056830 T3828 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056830 T3903 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056831 T3828 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056836 T3874 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 1056838 T3903 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 1056844 T3828 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 1057696 T3903 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1057707 T3903 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1057712 T3903 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1057728 T3903 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1057735 T3903 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1057739 T3903 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1057740 T3903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1057740 T3903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1057740 T3903 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1057741 T3903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1057742 T3903 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1057742 T3903 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1057742 T3903 oasc.SolrCore.<init> [testOverseerCol_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/data/
   [junit4]   2> 1057743 T3903 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f7cbeb
   [junit4]   2> 1057744 T3903 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/data
   [junit4]   2> 1057744 T3903 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/data/index/
   [junit4]   2> 1057744 T3903 oasc.SolrCore.initIndex WARN [testOverseerCol_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1057745 T3903 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1389967410441/testOverseerCol_shard2_replica1/data/index
   [junit4]   2> 1057745 T3903 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=87.029296875, floorSegmentMB=0.828125, forceMergeDeletesPctAllowed=0.4645446026141675, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6909869199532641
   [junit4]   2> 1057746 T3903 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@96de96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b0b105),segFN=segments_1,generation=1}
   [junit4]   2> 1057746 T3903 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1057750 T3903 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1057750 T3903 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1057751 T3903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1057751 T3903 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1057751 T3903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1057752 T3903 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1057752 T3903 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1057753 T3903 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1057753 T3903 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1057753 T3903 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1057754 T3903 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1057754 T3903 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1057755 T3903 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1057755 T3903 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1057756 T3903 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1057756 T3903 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1057757 T3903 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1057761 T3903 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1057773 T3903 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1057774 T3903 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1057774 T3903 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=97.6806640625, floorSegmentMB=0.9921875, forceMergeDeletesPctAllowed=29.270161109957726, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.37176894067279265
   [junit4]   2> 1057775 T3903 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@96de96 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b0b105),segFN=segments_1,generation=1}
   [junit4]   2> 1057776 T3903 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1057776 T3903 oass.SolrIndexSearcher.<init> Opening Searcher@12e1bbf main
   [junit4]   2> 1057779 T3937 oasc.SolrCore.registerSearcher [testOverseerCol_shard2_replica1] Registered new searcher Searcher@12e1bbf main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1057782 T3903 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard2_replica1
   [junit4]   2> 1057782 T3903 oasc.ZkController.register Register replica - core:testOverseerCol_shard2_replica1 address:http://127.0.0.1:34949 collection:testOverseerCol shard:shard2
   [junit4]   2> 1057784 T3903 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leader_elect/shard2/election
   [junit4]   2> 1057794 T3903 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1057796 T3834 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overse

[...truncated too long message...]

e.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data/index;done=false>>]
   [junit4]   2> 1075298 T3922 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data/index
   [junit4]   2> 1075298 T3922 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data;done=false>>]
   [junit4]   2> 1075299 T3922 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1389967411604/testOverseerCol_shard2_replica2/data
   [junit4]   2> 1075299 T3922 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leaders/shard2
   [junit4]   2> 1075299 T3922 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 1075299 T3922 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard2/election/91092903882391568-core_node6-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1075300 T3922 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1075300 T3922 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=ms_MY, timezone=Africa/Addis_Ababa
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=183725016,total=300417024
   [junit4]   2> NOTE: All tests run in this JVM: [FunctionTest, TestJoin, PrimitiveFieldTypeTest, TestCollapseQParserPlugin, SearchHandlerTest, FileBasedSpellCheckerTest, TestDocSet, LegacyHTMLStripCharFilterTest, TestSolrDeletionPolicy1, MoreLikeThisHandlerTest, DeleteShardTest, LukeRequestHandlerTest, SimpleFacetsTest, ExternalFileFieldSortTest, UnloadDistributedZkTest, SpellPossibilityIteratorTest, TestSearchPerf, TestSolrXMLSerializer, ParsingFieldUpdateProcessorsTest, ResourceLoaderTest, TestCollationFieldDocValues, TestSurroundQueryParser, HdfsLockFactoryTest, CoreMergeIndexesAdminHandlerTest, CoreAdminCreateDiscoverTest, TestJmxMonitoredMap, EnumFieldTest, QueryElevationComponentTest, TestDistributedMissingSort, PeerSyncTest, TestNoOpRegenerator, TestFunctionQuery, DistributedTermsComponentTest, TestRemoteStreaming, TestGroupingSearch, TestInfoStreamLogging, RegexBoostProcessorTest, DocValuesMissingTest, SolrCoreTest, TestQueryUtils, TestFaceting, TestCoreContainer, DistributedQueryElevationComponentTest, CircularListTest, ShardSplitTest, TestCustomSort, TestSolrQueryParserResource, TestValueSourceCache, SpatialFilterTest, HdfsBasicDistributedZkTest, TestDocumentBuilder, TestBadConfig, TermVectorComponentTest, TestCloudManagedSchemaAddField, TermsComponentTest, TestNonNRTOpen, SOLR749Test, BinaryUpdateRequestHandlerTest, TestLMJelinekMercerSimilarityFactory, BJQParserTest, StatelessScriptUpdateProcessorFactoryTest, ShowFileRequestHandlerTest, DocumentBuilderTest, TestRandomDVFaceting, TestSchemaSimilarityResource, IndexSchemaTest, DistributedSpellCheckComponentTest, TestExtendedDismaxParser, TestSimpleQParserPlugin, DOMUtilTest, OverseerCollectionProcessorTest, ZkControllerTest, SuggesterTest, BadComponentTest, AnalysisAfterCoreReloadTest, ScriptEngineTest, TestDynamicFieldCollectionResource, TestDynamicFieldResource, UUIDFieldTest, RecoveryZkTest, JsonLoaderTest, LeaderElectionTest, HdfsChaosMonkeySafeLeaderTest, SuggesterFSTTest, SynonymTokenizerTest, AutoCommitTest, SyncSliceTest, OverseerTest, ShardRoutingTest, BasicZkTest, ClusterStateUpdateTest, TestZkChroot, SliceStateUpdateTest, TermVectorComponentDistributedTest, TestRealTimeGet, HardAutoCommitTest, TestSolr4Spatial, SolrCmdDistributorTest, BadIndexSchemaTest, TestFiltering, SolrIndexSplitterTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, DocValuesTest, SuggesterTSTTest, TestTrie, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DistanceFunctionTest, DebugComponentTest, DisMaxRequestHandlerTest, TestWriterPerf, TestQueryTypes, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, TestArbitraryIndexDir, TestCollationField, ReturnFieldsTest, TestCSVResponseWriter, QueryParsingTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, TestLFUCache, TestQuerySenderListener, AlternateDirectoryTest, CopyFieldTest, TestStressRecovery, TestSystemIdResolver, TestSuggestSpellingConverter, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestLRUCache, TestUtils, SliceStateTest, SystemInfoHandlerTest, CursorPagingTest, OverseerRolesTest]
   [junit4] Completed on J0 in 28.20s, 1 test, 1 failure <<< FAILURES!

[...truncated 138 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 364 suites, 1596 tests, 1 failure, 34 ignored (6 assumptions)

Total time: 43 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure