You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/02/05 22:06:57 UTC

[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1887 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1887/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=4408, name=Overseer-91202134856761354-127.0.0.1:15375_ws%2Fw-n_0000000017, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4408, name=Overseer-91202134856761354-127.0.0.1:15375_ws%2Fw-n_0000000017, state=RUNNABLE, group=Overseer collection creation process.]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([1DC7C5615292A50E]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:209)
	at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:192)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:200)
	... 1 more




Build Log:
[...truncated 10557 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 2083538 T4227 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ws/w
   [junit4]   2> 2083545 T4227 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391634137823
   [junit4]   2> 2083547 T4227 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2083547 T4228 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2083648 T4227 oasc.ZkTestServer.run start zk server on port:15360
   [junit4]   2> 2083649 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2083654 T4234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f92269a name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2083654 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2083655 T4227 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2083665 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2083667 T4236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c7f8020 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2083667 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2083668 T4227 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2083676 T4227 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2083678 T4227 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2083680 T4227 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2083689 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2083690 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2083701 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2083702 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2083806 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2083807 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2083810 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2083810 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2083818 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2083819 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2083821 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2083822 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2083824 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2083824 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2083832 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2083833 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2083835 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2083836 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2083839 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2083840 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2083842 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2083843 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2084143 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2084148 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15363
   [junit4]   2> 2084149 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2084149 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2084150 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129
   [junit4]   2> 2084150 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/'
   [junit4]   2> 2084196 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/solr.xml
   [junit4]   2> 2084293 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2084294 T4227 oasc.CoreContainer.<init> New CoreContainer 462594701
   [junit4]   2> 2084295 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/]
   [junit4]   2> 2084297 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2084297 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2084298 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2084298 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2084299 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2084299 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2084300 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2084301 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2084301 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2084313 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2084314 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2084314 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2084315 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2084315 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2084317 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2084319 T4247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37ede51f name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2084320 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2084331 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2084333 T4249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fca55ac name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2084334 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2084336 T4227 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2084347 T4227 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2084350 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2084352 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15363_ws%2Fw
   [junit4]   2> 2084359 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15363_ws%2Fw
   [junit4]   2> 2084362 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2084364 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2084379 T4227 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:15363_ws%2Fw
   [junit4]   2> 2084380 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2084382 T4227 oasc.Overseer.start Overseer (id=91202134856761347-127.0.0.1:15363_ws%2Fw-n_0000000000) starting
   [junit4]   2> 2084395 T4227 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2084405 T4251 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2084406 T4227 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2084406 T4251 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2084408 T4227 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2084410 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2084413 T4250 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2084417 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129
   [junit4]   2> 2084420 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/conf/
   [junit4]   2> 2084599 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2084600 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2084601 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2084602 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2084605 T4254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aff32f7 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2084605 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2084606 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2084608 T4227 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2084907 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2084908 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2084912 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15367
   [junit4]   2> 2084913 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2084914 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2084914 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886
   [junit4]   2> 2084915 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/'
   [junit4]   2> 2084961 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/solr.xml
   [junit4]   2> 2085060 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2085061 T4227 oasc.CoreContainer.<init> New CoreContainer 1725079042
   [junit4]   2> 2085062 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/]
   [junit4]   2> 2085064 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2085064 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2085065 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2085066 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2085066 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2085067 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2085067 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2085068 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2085068 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2085080 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2085081 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2085081 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2085082 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2085083 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2085084 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2085112 T4265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d83ca0b name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2085112 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2085131 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2085133 T4267 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e0947f name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2085133 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2085144 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2086147 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15367_ws%2Fw
   [junit4]   2> 2086173 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15367_ws%2Fw
   [junit4]   2> 2086175 T4254 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2086176 T4267 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2086176 T4249 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2086190 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886
   [junit4]   2> 2086197 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/conf/
   [junit4]   2> 2086384 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2086385 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2086386 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2086690 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2086691 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2086695 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15370
   [junit4]   2> 2086696 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2086697 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2086698 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664
   [junit4]   2> 2086698 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/'
   [junit4]   2> 2086746 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/solr.xml
   [junit4]   2> 2086829 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2086830 T4227 oasc.CoreContainer.<init> New CoreContainer 6625381
   [junit4]   2> 2086830 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/]
   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2086833 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2086833 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2086835 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2086843 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2086843 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2086844 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2086844 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2086845 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2086846 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2086849 T4279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d248f16 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2086849 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2086868 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2086871 T4281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@401399e4 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2086871 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2086882 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2087885 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15370_ws%2Fw
   [junit4]   2> 2087913 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15370_ws%2Fw
   [junit4]   2> 2087916 T4249 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2087916 T4267 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2087916 T4254 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2087916 T4281 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2087931 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664
   [junit4]   2> 2088132 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/conf/
   [junit4]   2> 2088135 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2088136 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2088137 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2088445 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2088448 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2088451 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15375
   [junit4]   2> 2088451 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2088452 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2088453 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415
   [junit4]   2> 2088453 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/'
   [junit4]   2> 2088501 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/solr.xml
   [junit4]   2> 2088602 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2088603 T4227 oasc.CoreContainer.<init> New CoreContainer 175399980
   [junit4]   2> 2088604 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/]
   [junit4]   2> 2088606 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2088606 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2088607 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2088607 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2088608 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2088609 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2088609 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2088610 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2088610 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2088620 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2088621 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2088621 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2088622 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2088623 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2088624 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2088626 T4293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f5c8a10 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2088627 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2088629 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2088631 T4295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fb85ba3 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2088631 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2088642 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2089645 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15375_ws%2Fw
   [junit4]   2> 2089656 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15375_ws%2Fw
   [junit4]   2> 2089659 T4254 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2089659 T4281 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2089659 T4295 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2089659 T4267 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2089659 T4249 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2089677 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415
   [junit4]   2> 2089906 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/conf/
   [junit4]   2> 2089914 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2089914 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2089915 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2090250 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 2090252 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2090255 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15380
   [junit4]   2> 2090256 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2090257 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2090257 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193
   [junit4]   2> 2090258 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/'
   [junit4]   2> 2090304 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/solr.xml
   [junit4]   2> 2090381 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2090382 T4227 oasc.CoreContainer.<init> New CoreContainer 536865587
   [junit4]   2> 2090382 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/]
   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2090386 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2090386 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2090387 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2090393 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2090394 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2090394 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2090395 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2090395 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2090396 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090398 T4307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aaa71ae name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090399 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090401 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2090403 T4309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@554ea908 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2090403 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2090415 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2091417 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15380_ws%2Fw
   [junit4]   2> 2091419 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15380_ws%2Fw
   [junit4]   2> 2091440 T4254 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091440 T4309 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091440 T4267 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091440 T4249 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091440 T4281 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091440 T4295 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2091447 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193
   [junit4]   2> 2091738 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/conf/
   [junit4]   2> 2091740 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2091741 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2091742 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2092034 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 2092036 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2092040 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15396
   [junit4]   2> 2092040 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2092041 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2092042 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019
   [junit4]   2> 2092042 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/'
   [junit4]   2> 2092088 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/solr.xml
   [junit4]   2> 2092186 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2092187 T4227 oasc.CoreContainer.<init> New CoreContainer 439564509
   [junit4]   2> 2092187 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/]
   [junit4]   2> 2092189 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2092190 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2092190 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2092191 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2092191 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2092192 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2092193 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2092193 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2092194 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2092205 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2092206 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2092206 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2092207 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2092207 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2092209 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2092238 T4321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18da429b name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2092239 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2092253 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2092255 T4323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c6ba5f6 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2092256 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2092272 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2093275 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15396_ws%2Fw
   [junit4]   2> 2093292 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15396_ws%2Fw
   [junit4]   2> 2093296 T4267 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4295 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4309 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4249 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4323 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4281 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093296 T4254 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2093310 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019
   [junit4]   2> 2093493 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/conf/
   [junit4]   2> 2093495 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2093496 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2093497 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2093790 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 2093792 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2093795 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15399
   [junit4]   2> 2093796 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2093797 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2093797 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774
   [junit4]   2> 2093798 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/'
   [junit4]   2> 2093842 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/solr.xml
   [junit4]   2> 2093933 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2093934 T4227 oasc.CoreContainer.<init> New CoreContainer 1253563338
   [junit4]   2> 2093935 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/]
   [junit4]   2> 2093937 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2093937 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2093938 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2093938 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2093939 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2093939 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2093940 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2093940 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2093941 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2093952 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2093952 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2093953 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2093953 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
   [junit4]   2> 2093954 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2093955 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093958 T4335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cbed118 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093958 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093962 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2093973 T4337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ef8182b name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2093973 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2093978 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2094981 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15399_ws%2Fw
   [junit4]   2> 2094992 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15399_ws%2Fw
   [junit4]   2> 2094995 T4254 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4249 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4281 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4337 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4295 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4267 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4323 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2094996 T4309 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2095013 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774
   [junit4]   2> 2095023 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/conf/
   [junit4]   2> 2095375 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2095376 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 2095377 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2095378 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2095380 T4340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6081042f name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2095381 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2095382 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2095385 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2095387 T4342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6722d0ba name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2095387 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2095388 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2095392 T4257 oasha.CollectionsHandler.handleCreateAction Creating Collection : maxShardsPerNode=2&version=2&numShards=4&action=CREATE&replicationFactor=2&wt=javabin&name=testOverseerCol
   [junit4]   2> 2095394 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095395 T4251 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2095395 T4251 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2095396 T4251 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2095396 T4251 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 2095397 T4251 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 2095413 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095414 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2095415 T4250 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 2095415 T4250 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2095418 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095428 T4337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095428 T4267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095514 T4251 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 2095514 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:15363_ws%2Fw
   [junit4]   2> 2095515 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:15370_ws%2Fw
   [junit4]   2> 2095517 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:15399_ws%2Fw
   [junit4]   2> 2095517 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:15396_ws%2Fw
   [junit4]   2> 2095518 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:15380_ws%2Fw
   [junit4]   2> 2095519 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:15367_ws%2Fw
   [junit4]   2> 2095520 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:15375_ws%2Fw
   [junit4]   2> 2095521 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:15363_ws%2Fw
   [junit4]   2> 2095521 T4271 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 2095521 T4239 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 2095523 T4327 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 2095522 T4313 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 2095524 T4299 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 2095524 T4240 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 2095524 T4285 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 2095524 T4271 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095524 T4258 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 2095524 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095525 T4327 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095525 T4239 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095535 T4313 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095536 T4299 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095536 T4285 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095536 T4258 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095536 T4240 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2095537 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2095538 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15370/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15370_ws%2Fw",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095550 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095551 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15363/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15363_ws%2Fw",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095560 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095561 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15399/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15399_ws%2Fw",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095565 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095565 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15396/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15396_ws%2Fw",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095574 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095575 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15380/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15380_ws%2Fw",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095583 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095584 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15363/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15363_ws%2Fw",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095586 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095587 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15375/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15375_ws%2Fw",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095596 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095596 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15367/ws/w",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:15367_ws%2Fw",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2095599 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2095702 T4267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2095702 T4295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2096527 T4271 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 2096528 T4239 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 2096528 T4327 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 2096530 T4271 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/testOverseerCol_shard1_replica2
   [junit4]   2> 2096530 T4239 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1
   [junit4]   2> 2096530 T4271 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096530 T4239 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096531 T4327 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/testOverseerCol_shard2_replica1
   [junit4]   2> 2096531 T4239 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096531 T4271 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096532 T4239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096532 T4327 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096532 T4271 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096533 T4327 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096533 T4239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/'
   [junit4]   2> 2096534 T4271 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/testOverseerCol_shard1_replica2/'
   [junit4]   2> 2096534 T4327 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096536 T4313 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 2096536 T4327 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/testOverseerCol_shard2_replica1/'
   [junit4]   2> 2096537 T4299 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 2096538 T4258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 2096538 T4313 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/testOverseerCol_shard2_replica2
   [junit4]   2> 2096539 T4240 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 2096538 T4285 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 2096539 T4313 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096540 T4299 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/testOverseerCol_shard3_replica1
   [junit4]   2> 2096540 T4313 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096541 T4258 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/testOverseerCol_shard3_replica2
   [junit4]   2> 2096540 T4299 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096541 T4285 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/testOverseerCol_shard4_replica1
   [junit4]   2> 2096542 T4299 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096543 T4299 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096541 T4258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096541 T4240 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard4_replica2
   [junit4]   2> 2096544 T4258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096541 T4313 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096544 T4258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096544 T4240 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096544 T4299 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/testOverseerCol_shard3_replica1/'
   [junit4]   2> 2096542 T4285 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2096546 T4240 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096547 T4285 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2096546 T4258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/testOverseerCol_shard3_replica2/'
   [junit4]   2> 2096545 T4313 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/testOverseerCol_shard2_replica2/'
   [junit4]   2> 2096547 T4285 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096547 T4240 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2096550 T4285 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/testOverseerCol_shard4_replica1/'
   [junit4]   2> 2096550 T4240 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard4_replica2/'
   [junit4]   2> 2096820 T4258 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096821 T4327 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096821 T4271 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096822 T4313 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096823 T4239 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096827 T4299 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096829 T4285 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096829 T4240 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2096919 T4327 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096921 T4271 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096926 T4285 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096929 T4313 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096940 T4258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096945 T4239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096980 T4239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2096980 T4299 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096980 T4240 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2096982 T4240 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2096989 T4239 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 2096994 T4240 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 2097021 T4327 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097022 T4271 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097027 T4285 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097031 T4313 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097035 T4327 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 2097035 T4271 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 2097037 T4285 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 2097041 T4313 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 2097042 T4258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097057 T4258 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 2097081 T4299 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2097097 T4299 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 2098374 T4239 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098375 T4239 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098377 T4239 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2098394 T4239 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2098400 T4239 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2098421 T4239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2098434 T4239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2098444 T4239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2098446 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2098447 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2098447 T4239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2098449 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2098450 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2098450 T4239 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2098451 T4239 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/
   [junit4]   2> 2098451 T4239 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ea3e16
   [junit4]   2> 2098452 T4239 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data
   [junit4]   2> 2098453 T4239 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index/
   [junit4]   2> 2098453 T4239 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2098454 T4239 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index
   [junit4]   2> 2098455 T4239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=89.447265625, floorSegmentMB=0.263671875, forceMergeDeletesPctAllowed=9.418735929157263, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7370034703480962
   [junit4]   2> 2098456 T4239 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3d4b160c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@585a04d3),segFN=segments_1,generation=1}
   [junit4]   2> 2098456 T4239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2098462 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2098464 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2098464 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2098465 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2098465 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2098466 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2098467 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2098468 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2098468 T4239 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2098469 T4239 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2098470 T4239 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2098471 T4239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2098471 T4239 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2098472 T4239 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2098473 T4239 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2098474 T4239 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2098507 T4285 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098508 T4285 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098510 T4285 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2098515 T4239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2098525 T4239 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2098526 T4239 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2098527 T4239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=44.1904296875, floorSegmentMB=0.5068359375, forceMergeDeletesPctAllowed=10.924763052934182, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2098528 T4285 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2098529 T4240 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098528 T4239 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3d4b160c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@585a04d3),segFN=segments_1,generation=1}
   [junit4]   2> 2098530 T4240 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098531 T4239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2098532 T4239 oass.SolrIndexSearcher.<init> Opening Searcher@79960325 main
   [junit4]   2> 2098532 T4240 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2098575 T4285 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2098576 T4351 oasc.SolrCore.registerSearcher [testOverseerCol_shard1_replica1] Registered new searcher Searcher@79960325 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2098580 T4239 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard1_replica1
   [junit4]   2> 2098581 T4239 oasc.ZkController.register Register replica - core:testOverseerCol_shard1_replica1 address:http://127.0.0.1:15363/ws/w collection:testOverseerCol shard:shard1
   [junit4]   2> 2098581 T4313 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098582 T4313 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098584 T4313 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2098586 T4239 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leader_elect/shard1/election
   [junit4]   2> 2098586 T4240 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2098594 T4240 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2098599 T4313 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2098605 T4313 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2098629 T4239 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2098636 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2098637 T4258 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098638 T4285 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2098638 T4258 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2098639 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2098637 T4239 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
   [junit4]   2> 2098637 T4249 oasc.DistributedQue

[...truncated too long message...]

ion.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 2107564 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard2/election/91202134856761360-core_node3-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2107564 T4337 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2107565 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2107565 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=sq_AL, timezone=Europe/Belfast
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=142932688,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestFaceting, TestCollationFieldDocValues, DocValuesMultiTest, QueryElevationComponentTest, IndexBasedSpellCheckerTest, CollectionsAPIDistributedZkTest, EchoParamsTest, TestCSVLoader, TestSystemIdResolver, TestSchemaSimilarityResource, TestCoreDiscovery, SpellCheckCollatorTest, CustomCollectionTest, IndexSchemaTest, TestExtendedDismaxParser, TestStressRecovery, TimeZoneUtilsTest, TestSchemaNameResource, SliceStateTest, ResponseLogComponentTest, LukeRequestHandlerTest, TestConfig, HdfsRecoveryZkTest, JSONWriterTest, TestStressVersions, TestTrie, CachingDirectoryFactoryTest, DocumentAnalysisRequestHandlerTest, OpenExchangeRatesOrgProviderTest, TestMergePolicyConfig, TestSolrXml, ParsingFieldUpdateProcessorsTest, TestNoOpRegenerator, TestSuggestSpellingConverter, TestSerializedLuceneMatchVersion, TestLMJelinekMercerSimilarityFactory, TestStandardQParsers, CacheHeaderTest, BadIndexSchemaTest, TestJoin, SoftAutoCommitTest, SyncSliceTest, TestRandomFaceting, TestDefaultSimilarityFactory, PingRequestHandlerTest, TestAnalyzedSuggestions, RequiredFieldsTest, BadComponentTest, SynonymTokenizerTest, TestIndexingPerformance, TestFuzzyAnalyzedSuggestions, TestDynamicFieldResource, HdfsBasicDistributedZk2Test, TestFiltering, SuggestComponentTest, HdfsChaosMonkeySafeLeaderTest, CoreAdminHandlerTest, TestLazyCores, SchemaVersionSpecificBehaviorTest, CoreAdminCreateDiscoverTest, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, SuggesterTSTTest, NoCacheHeaderTest, PrimUtilsTest, SystemInfoHandlerTest, TestSolr4Spatial, TestLRUCache, AssignTest, DistanceFunctionTest, InfoHandlerTest, ZkSolrClientTest, DistributedQueryElevationComponentTest, TestGroupingSearch, TestSearchPerf, TestInfoStreamLogging, ShowFileRequestHandlerTest, TestManagedSchemaFieldResource, XmlUpdateRequestHandlerTest, SuggesterWFSTTest, TestNonNRTOpen, TestQueryUtils, TestManagedSchema, BasicDistributedZk2Test, BasicZkTest, TriLevelCompositeIdRoutingTest, TestRandomDVFaceting, FullSolrCloudDistribCmdsTest, TestCloudManagedSchemaAddField, TestReload, OverseerTest, OverseerRolesTest]
   [junit4] Completed on J0 in 24.88s, 1 test, 1 error <<< FAILURES!

[...truncated 498 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:459: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:941: There were test failures: 358 suites, 1552 tests, 1 error, 35 ignored (7 assumptions)

Total time: 66 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1887 - Still Failing

Posted by Mark Miller <ma...@gmail.com>.
It seems this test must somehow trigger two Overseers running at the same time.

- Mark

http://about.me/markrmiller

On Feb 5, 2014, at 4:06 PM, Apache Jenkins Server <je...@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1887/
> 
> 1 tests failed.
> REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch
> 
> Error Message:
> Captured an uncaught exception in thread: Thread[id=4408, name=Overseer-91202134856761354-127.0.0.1:15375_ws%2Fw-n_0000000017, state=RUNNABLE, group=Overseer collection creation process.]
> 
> Stack Trace:
> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4408, name=Overseer-91202134856761354-127.0.0.1:15375_ws%2Fw-n_0000000017, state=RUNNABLE, group=Overseer collection creation process.]
> Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
> 	at __randomizedtesting.SeedInfo.seed([1DC7C5615292A50E]:0)
> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:209)
> 	at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
> 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:192)
> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:200)
> 	... 1 more
> 
> 
> 
> 
> Build Log:
> [...truncated 10557 lines...]
>   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
>   [junit4]   2> 2083538 T4227 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ws/w
>   [junit4]   2> 2083545 T4227 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391634137823
>   [junit4]   2> 2083547 T4227 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 2083547 T4228 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 2083648 T4227 oasc.ZkTestServer.run start zk server on port:15360
>   [junit4]   2> 2083649 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2083654 T4234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f92269a name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2083654 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2083655 T4227 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 2083665 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2083667 T4236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c7f8020 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2083667 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2083668 T4227 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 2083676 T4227 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 2083678 T4227 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 2083680 T4227 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 2083689 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 2083690 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 2083701 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
>   [junit4]   2> 2083702 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 2083806 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 2083807 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 2083810 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 2083810 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 2083818 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 2083819 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 2083821 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 2083822 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 2083824 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 2083824 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 2083832 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 2083833 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 2083835 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 2083836 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 2083839 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 2083840 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 2083842 T4227 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 2083843 T4227 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> 2084143 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2084148 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15363
>   [junit4]   2> 2084149 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2084149 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2084150 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129
>   [junit4]   2> 2084150 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/'
>   [junit4]   2> 2084196 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/solr.xml
>   [junit4]   2> 2084293 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2084294 T4227 oasc.CoreContainer.<init> New CoreContainer 462594701
>   [junit4]   2> 2084295 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/]
>   [junit4]   2> 2084297 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2084297 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2084298 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2084298 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2084299 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2084299 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2084300 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2084301 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2084301 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2084313 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2084314 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2084314 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2084315 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2084315 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2084317 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2084319 T4247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37ede51f name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2084320 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2084331 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2084333 T4249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fca55ac name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2084334 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2084336 T4227 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 2084347 T4227 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 2084350 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 2084352 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15363_ws%2Fw
>   [junit4]   2> 2084359 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15363_ws%2Fw
>   [junit4]   2> 2084362 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect
>   [junit4]   2> 2084364 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 2084379 T4227 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:15363_ws%2Fw
>   [junit4]   2> 2084380 T4227 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 2084382 T4227 oasc.Overseer.start Overseer (id=91202134856761347-127.0.0.1:15363_ws%2Fw-n_0000000000) starting
>   [junit4]   2> 2084395 T4227 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 2084405 T4251 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 2084406 T4227 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 2084406 T4251 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>   [junit4]   2> 2084408 T4227 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 2084410 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2084413 T4250 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 2084417 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129
>   [junit4]   2> 2084420 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/conf/
>   [junit4]   2> 2084599 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2084600 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2084601 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2084602 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2084605 T4254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aff32f7 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2084605 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2084606 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2084608 T4227 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 2084907 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>   [junit4]   2> 2084908 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2084912 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15367
>   [junit4]   2> 2084913 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2084914 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2084914 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886
>   [junit4]   2> 2084915 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/'
>   [junit4]   2> 2084961 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/solr.xml
>   [junit4]   2> 2085060 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2085061 T4227 oasc.CoreContainer.<init> New CoreContainer 1725079042
>   [junit4]   2> 2085062 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/]
>   [junit4]   2> 2085064 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2085064 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2085065 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2085066 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2085066 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2085067 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2085067 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2085068 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2085068 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2085080 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2085081 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2085081 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2085082 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2085083 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2085084 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2085112 T4265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d83ca0b name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2085112 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2085131 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2085133 T4267 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e0947f name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2085133 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2085144 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2086147 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15367_ws%2Fw
>   [junit4]   2> 2086173 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15367_ws%2Fw
>   [junit4]   2> 2086175 T4254 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 2086176 T4267 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 2086176 T4249 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 2086190 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886
>   [junit4]   2> 2086197 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/conf/
>   [junit4]   2> 2086384 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2086385 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2086386 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2086690 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>   [junit4]   2> 2086691 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2086695 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15370
>   [junit4]   2> 2086696 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2086697 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2086698 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664
>   [junit4]   2> 2086698 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/'
>   [junit4]   2> 2086746 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/solr.xml
>   [junit4]   2> 2086829 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2086830 T4227 oasc.CoreContainer.<init> New CoreContainer 6625381
>   [junit4]   2> 2086830 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/]
>   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2086832 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2086833 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2086833 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2086834 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2086835 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2086843 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2086843 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2086844 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2086844 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2086845 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2086846 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2086849 T4279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d248f16 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2086849 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2086868 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2086871 T4281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@401399e4 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2086871 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2086882 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2087885 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15370_ws%2Fw
>   [junit4]   2> 2087913 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15370_ws%2Fw
>   [junit4]   2> 2087916 T4249 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 2087916 T4267 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 2087916 T4254 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 2087916 T4281 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 2087931 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664
>   [junit4]   2> 2088132 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/conf/
>   [junit4]   2> 2088135 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2088136 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2088137 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2088445 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
>   [junit4]   2> 2088448 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2088451 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15375
>   [junit4]   2> 2088451 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2088452 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2088453 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415
>   [junit4]   2> 2088453 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/'
>   [junit4]   2> 2088501 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/solr.xml
>   [junit4]   2> 2088602 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2088603 T4227 oasc.CoreContainer.<init> New CoreContainer 175399980
>   [junit4]   2> 2088604 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/]
>   [junit4]   2> 2088606 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2088606 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2088607 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2088607 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2088608 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2088609 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2088609 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2088610 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2088610 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2088620 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2088621 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2088621 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2088622 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2088623 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2088624 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2088626 T4293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f5c8a10 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2088627 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2088629 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2088631 T4295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fb85ba3 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2088631 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2088642 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2089645 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15375_ws%2Fw
>   [junit4]   2> 2089656 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15375_ws%2Fw
>   [junit4]   2> 2089659 T4254 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 2089659 T4281 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 2089659 T4295 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 2089659 T4267 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 2089659 T4249 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 2089677 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415
>   [junit4]   2> 2089906 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/conf/
>   [junit4]   2> 2089914 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2089914 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2089915 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2090250 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
>   [junit4]   2> 2090252 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2090255 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15380
>   [junit4]   2> 2090256 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2090257 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2090257 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193
>   [junit4]   2> 2090258 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/'
>   [junit4]   2> 2090304 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/solr.xml
>   [junit4]   2> 2090381 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2090382 T4227 oasc.CoreContainer.<init> New CoreContainer 536865587
>   [junit4]   2> 2090382 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/]
>   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2090384 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2090385 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2090386 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2090386 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2090387 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2090393 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2090394 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2090394 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2090395 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2090395 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2090396 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2090398 T4307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aaa71ae name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2090399 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2090401 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2090403 T4309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@554ea908 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2090403 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2090415 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2091417 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15380_ws%2Fw
>   [junit4]   2> 2091419 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15380_ws%2Fw
>   [junit4]   2> 2091440 T4254 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091440 T4309 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091440 T4267 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091440 T4249 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091440 T4281 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091440 T4295 oascc.ZkStateReader$3.process Updating live nodes... (5)
>   [junit4]   2> 2091447 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193
>   [junit4]   2> 2091738 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/conf/
>   [junit4]   2> 2091740 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2091741 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2091742 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2092034 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
>   [junit4]   2> 2092036 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2092040 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15396
>   [junit4]   2> 2092040 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2092041 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2092042 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019
>   [junit4]   2> 2092042 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/'
>   [junit4]   2> 2092088 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/solr.xml
>   [junit4]   2> 2092186 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2092187 T4227 oasc.CoreContainer.<init> New CoreContainer 439564509
>   [junit4]   2> 2092187 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/]
>   [junit4]   2> 2092189 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2092190 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2092190 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2092191 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2092191 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2092192 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2092193 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2092193 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2092194 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2092205 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2092206 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2092206 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2092207 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2092207 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2092209 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2092238 T4321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18da429b name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2092239 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2092253 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2092255 T4323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c6ba5f6 name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2092256 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2092272 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2093275 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15396_ws%2Fw
>   [junit4]   2> 2093292 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15396_ws%2Fw
>   [junit4]   2> 2093296 T4267 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4295 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4309 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4249 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4323 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4281 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093296 T4254 oascc.ZkStateReader$3.process Updating live nodes... (6)
>   [junit4]   2> 2093310 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019
>   [junit4]   2> 2093493 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/conf/
>   [junit4]   2> 2093495 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2093496 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2093497 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2093790 T4227 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
>   [junit4]   2> 2093792 T4227 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 2093795 T4227 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15399
>   [junit4]   2> 2093796 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 2093797 T4227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 2093797 T4227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774
>   [junit4]   2> 2093798 T4227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/'
>   [junit4]   2> 2093842 T4227 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/solr.xml
>   [junit4]   2> 2093933 T4227 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
>   [junit4]   2> 2093934 T4227 oasc.CoreContainer.<init> New CoreContainer 1253563338
>   [junit4]   2> 2093935 T4227 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/]
>   [junit4]   2> 2093937 T4227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 2093937 T4227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 2093938 T4227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 2093938 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 2093939 T4227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 2093939 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 2093940 T4227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 2093940 T4227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 2093941 T4227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 2093952 T4227 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 2093952 T4227 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 2093953 T4227 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 2093953 T4227 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:15360/solr
>   [junit4]   2> 2093954 T4227 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 2093955 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2093958 T4335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cbed118 name:ZooKeeperConnection Watcher:127.0.0.1:15360 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2093958 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2093962 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2093973 T4337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ef8182b name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2093973 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2093978 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2094981 T4227 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15399_ws%2Fw
>   [junit4]   2> 2094992 T4227 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15399_ws%2Fw
>   [junit4]   2> 2094995 T4254 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4249 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4281 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4337 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4295 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4267 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4323 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2094996 T4309 oascc.ZkStateReader$3.process Updating live nodes... (7)
>   [junit4]   2> 2095013 T4227 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774
>   [junit4]   2> 2095023 T4227 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/conf/
>   [junit4]   2> 2095375 T4227 oasc.CorePropertiesLocator.discover Found 1 core definitions
>   [junit4]   2> 2095376 T4227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
>   [junit4]   2> 2095377 T4227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 2095378 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2095380 T4340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6081042f name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2095381 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2095382 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2095385 T4227 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 2095387 T4342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6722d0ba name:ZooKeeperConnection Watcher:127.0.0.1:15360/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 2095387 T4227 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 2095388 T4227 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 2095392 T4257 oasha.CollectionsHandler.handleCreateAction Creating Collection : maxShardsPerNode=2&version=2&numShards=4&action=CREATE&replicationFactor=2&wt=javabin&name=testOverseerCol
>   [junit4]   2> 2095394 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095395 T4251 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"testOverseerCol",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "maxShardsPerNode":"2"}
>   [junit4]   2> 2095395 T4251 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
>   [junit4]   2> 	  "operation":"createcollection",
>   [junit4]   2> 	  "fromApi":"true",
>   [junit4]   2> 	  "name":"testOverseerCol",
>   [junit4]   2> 	  "replicationFactor":"2",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "maxShardsPerNode":"2"}
>   [junit4]   2> 2095396 T4251 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
>   [junit4]   2> 2095396 T4251 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
>   [junit4]   2> 2095397 T4251 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
>   [junit4]   2> 2095413 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095414 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 2095415 T4250 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
>   [junit4]   2> 2095415 T4250 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
>   [junit4]   2> 2095418 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095428 T4337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095428 T4267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095514 T4251 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
>   [junit4]   2> 2095514 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:15363_ws%2Fw
>   [junit4]   2> 2095515 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:15370_ws%2Fw
>   [junit4]   2> 2095517 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:15399_ws%2Fw
>   [junit4]   2> 2095517 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:15396_ws%2Fw
>   [junit4]   2> 2095518 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:15380_ws%2Fw
>   [junit4]   2> 2095519 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:15367_ws%2Fw
>   [junit4]   2> 2095520 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:15375_ws%2Fw
>   [junit4]   2> 2095521 T4251 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:15363_ws%2Fw
>   [junit4]   2> 2095521 T4271 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
>   [junit4]   2> 2095521 T4239 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
>   [junit4]   2> 2095523 T4327 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
>   [junit4]   2> 2095522 T4313 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
>   [junit4]   2> 2095524 T4299 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
>   [junit4]   2> 2095524 T4240 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
>   [junit4]   2> 2095524 T4285 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
>   [junit4]   2> 2095524 T4271 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095524 T4258 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
>   [junit4]   2> 2095524 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095525 T4327 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095525 T4239 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095535 T4313 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095536 T4299 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095536 T4285 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095536 T4258 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095536 T4240 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 2095537 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 2095538 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15370/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15370_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095550 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095551 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15363/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15363_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095560 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095561 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15399/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15399_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095565 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095565 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15396/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15396_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095574 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095575 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15380/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15380_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard3",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095583 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095584 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15363/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15363_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard4",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095586 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095587 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15375/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15375_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard4",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095596 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095596 T4250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:15367/ws/w",
>   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:15367_ws%2Fw",
>   [junit4]   2> 	  "shard":"shard3",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"testOverseerCol",
>   [junit4]   2> 	  "numShards":"4",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 2095599 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2095702 T4267 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4337 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4340 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4249 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2095702 T4295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
>   [junit4]   2> 2096527 T4271 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
>   [junit4]   2> 2096528 T4239 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
>   [junit4]   2> 2096528 T4327 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
>   [junit4]   2> 2096530 T4271 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/testOverseerCol_shard1_replica2
>   [junit4]   2> 2096530 T4239 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1
>   [junit4]   2> 2096530 T4271 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096530 T4239 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096531 T4327 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/testOverseerCol_shard2_replica1
>   [junit4]   2> 2096531 T4239 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096531 T4271 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096532 T4239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096532 T4327 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096532 T4271 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096533 T4327 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096533 T4239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/'
>   [junit4]   2> 2096534 T4271 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391634140664/testOverseerCol_shard1_replica2/'
>   [junit4]   2> 2096534 T4327 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096536 T4313 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
>   [junit4]   2> 2096536 T4327 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391634147774/testOverseerCol_shard2_replica1/'
>   [junit4]   2> 2096537 T4299 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
>   [junit4]   2> 2096538 T4258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
>   [junit4]   2> 2096538 T4313 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/testOverseerCol_shard2_replica2
>   [junit4]   2> 2096539 T4240 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
>   [junit4]   2> 2096538 T4285 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
>   [junit4]   2> 2096539 T4313 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096540 T4299 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/testOverseerCol_shard3_replica1
>   [junit4]   2> 2096540 T4313 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096541 T4258 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/testOverseerCol_shard3_replica2
>   [junit4]   2> 2096540 T4299 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096541 T4285 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/testOverseerCol_shard4_replica1
>   [junit4]   2> 2096542 T4299 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096543 T4299 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096541 T4258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096541 T4240 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard4_replica2
>   [junit4]   2> 2096544 T4258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096541 T4313 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096544 T4258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096544 T4240 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096544 T4299 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391634144193/testOverseerCol_shard3_replica1/'
>   [junit4]   2> 2096542 T4285 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
>   [junit4]   2> 2096546 T4240 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096547 T4285 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 2096546 T4258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391634138886/testOverseerCol_shard3_replica2/'
>   [junit4]   2> 2096545 T4313 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391634146019/testOverseerCol_shard2_replica2/'
>   [junit4]   2> 2096547 T4285 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096547 T4240 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
>   [junit4]   2> 2096550 T4285 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391634142415/testOverseerCol_shard4_replica1/'
>   [junit4]   2> 2096550 T4240 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard4_replica2/'
>   [junit4]   2> 2096820 T4258 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096821 T4327 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096821 T4271 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096822 T4313 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096823 T4239 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096827 T4299 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096829 T4285 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096829 T4240 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 2096919 T4327 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096921 T4271 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096926 T4285 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096929 T4313 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096940 T4258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096945 T4239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096980 T4239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2096980 T4299 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096980 T4240 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 2096982 T4240 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2096989 T4239 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
>   [junit4]   2> 2096994 T4240 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
>   [junit4]   2> 2097021 T4327 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097022 T4271 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097027 T4285 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097031 T4313 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097035 T4327 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
>   [junit4]   2> 2097035 T4271 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
>   [junit4]   2> 2097037 T4285 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
>   [junit4]   2> 2097041 T4313 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
>   [junit4]   2> 2097042 T4258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097057 T4258 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
>   [junit4]   2> 2097081 T4299 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 2097097 T4299 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
>   [junit4]   2> 2098374 T4239 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098375 T4239 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098377 T4239 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 2098394 T4239 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 2098400 T4239 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 2098421 T4239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 2098434 T4239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 2098444 T4239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 2098446 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 2098447 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 2098447 T4239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
>   [junit4]   2> 2098449 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
>   [junit4]   2> 2098450 T4239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
>   [junit4]   2> 2098450 T4239 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 2098451 T4239 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/
>   [junit4]   2> 2098451 T4239 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@68ea3e16
>   [junit4]   2> 2098452 T4239 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data
>   [junit4]   2> 2098453 T4239 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index/
>   [junit4]   2> 2098453 T4239 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 2098454 T4239 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391634138129/testOverseerCol_shard1_replica1/data/index
>   [junit4]   2> 2098455 T4239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=89.447265625, floorSegmentMB=0.263671875, forceMergeDeletesPctAllowed=9.418735929157263, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7370034703480962
>   [junit4]   2> 2098456 T4239 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3d4b160c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@585a04d3),segFN=segments_1,generation=1}
>   [junit4]   2> 2098456 T4239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 2098462 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 2098464 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 2098464 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 2098465 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 2098465 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 2098466 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 2098467 T4239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 2098468 T4239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 2098468 T4239 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 2098469 T4239 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 2098470 T4239 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 2098471 T4239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 2098471 T4239 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 2098472 T4239 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 2098473 T4239 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 2098474 T4239 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 2098507 T4285 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098508 T4285 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098510 T4285 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 2098515 T4239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 2098525 T4239 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 2098526 T4239 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 2098527 T4239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=44.1904296875, floorSegmentMB=0.5068359375, forceMergeDeletesPctAllowed=10.924763052934182, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>   [junit4]   2> 2098528 T4285 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 2098529 T4240 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098528 T4239 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3d4b160c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@585a04d3),segFN=segments_1,generation=1}
>   [junit4]   2> 2098530 T4240 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098531 T4239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 2098532 T4239 oass.SolrIndexSearcher.<init> Opening Searcher@79960325 main
>   [junit4]   2> 2098532 T4240 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 2098575 T4285 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 2098576 T4351 oasc.SolrCore.registerSearcher [testOverseerCol_shard1_replica1] Registered new searcher Searcher@79960325 main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 2098580 T4239 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard1_replica1
>   [junit4]   2> 2098581 T4239 oasc.ZkController.register Register replica - core:testOverseerCol_shard1_replica1 address:http://127.0.0.1:15363/ws/w collection:testOverseerCol shard:shard1
>   [junit4]   2> 2098581 T4313 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098582 T4313 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098584 T4313 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>   [junit4]   2> 2098586 T4239 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leader_elect/shard1/election
>   [junit4]   2> 2098586 T4240 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 2098594 T4240 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 2098599 T4313 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 2098605 T4313 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 2098629 T4239 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 2098636 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 2098637 T4258 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098638 T4285 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 2098638 T4258 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
>   [junit4]   2> 2098639 T4250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 2098637 T4239 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
>   [junit4]   2> 2098637 T4249 oasc.DistributedQue
> 
> [...truncated too long message...]
> 
> ion.java:51)
>   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
>   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
>   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
>   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
>   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 	
>   [junit4]   2> 2107564 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard2/election/91202134856761360-core_node3-n_0000000001
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
>   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
>   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 2107564 T4337 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
>   [junit4]   2> 2107565 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> 2107565 T4337 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
>   [junit4]   2> 
>   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=sq_AL, timezone=Europe/Belfast
>   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=142932688,total=536870912
>   [junit4]   2> NOTE: All tests run in this JVM: [TestFaceting, TestCollationFieldDocValues, DocValuesMultiTest, QueryElevationComponentTest, IndexBasedSpellCheckerTest, CollectionsAPIDistributedZkTest, EchoParamsTest, TestCSVLoader, TestSystemIdResolver, TestSchemaSimilarityResource, TestCoreDiscovery, SpellCheckCollatorTest, CustomCollectionTest, IndexSchemaTest, TestExtendedDismaxParser, TestStressRecovery, TimeZoneUtilsTest, TestSchemaNameResource, SliceStateTest, ResponseLogComponentTest, LukeRequestHandlerTest, TestConfig, HdfsRecoveryZkTest, JSONWriterTest, TestStressVersions, TestTrie, CachingDirectoryFactoryTest, DocumentAnalysisRequestHandlerTest, OpenExchangeRatesOrgProviderTest, TestMergePolicyConfig, TestSolrXml, ParsingFieldUpdateProcessorsTest, TestNoOpRegenerator, TestSuggestSpellingConverter, TestSerializedLuceneMatchVersion, TestLMJelinekMercerSimilarityFactory, TestStandardQParsers, CacheHeaderTest, BadIndexSchemaTest, TestJoin, SoftAutoCommitTest, SyncSliceTest, TestRandomFaceting, TestDefaultSimilarityFactory, PingRequestHandlerTest, TestAnalyzedSuggestions, RequiredFieldsTest, BadComponentTest, SynonymTokenizerTest, TestIndexingPerformance, TestFuzzyAnalyzedSuggestions, TestDynamicFieldResource, HdfsBasicDistributedZk2Test, TestFiltering, SuggestComponentTest, HdfsChaosMonkeySafeLeaderTest, CoreAdminHandlerTest, TestLazyCores, SchemaVersionSpecificBehaviorTest, CoreAdminCreateDiscoverTest, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, SuggesterTSTTest, NoCacheHeaderTest, PrimUtilsTest, SystemInfoHandlerTest, TestSolr4Spatial, TestLRUCache, AssignTest, DistanceFunctionTest, InfoHandlerTest, ZkSolrClientTest, DistributedQueryElevationComponentTest, TestGroupingSearch, TestSearchPerf, TestInfoStreamLogging, ShowFileRequestHandlerTest, TestManagedSchemaFieldResource, XmlUpdateRequestHandlerTest, SuggesterWFSTTest, TestNonNRTOpen, TestQueryUtils, TestManagedSchema, BasicDistributedZk2Test, BasicZkTest, TriLevelCompositeIdRoutingTest, TestRandomDVFaceting, FullSolrCloudDistribCmdsTest, TestCloudManagedSchemaAddField, TestReload, OverseerTest, OverseerRolesTest]
>   [junit4] Completed on J0 in 24.88s, 1 test, 1 error <<< FAILURES!
> 
> [...truncated 498 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:459: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:439: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:491: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1308: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:941: There were test failures: 358 suites, 1552 tests, 1 error, 35 ignored (7 assumptions)
> 
> Total time: 66 minutes 19 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org