You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/01 12:10:06 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1283 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1283/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=10394, name=Overseer-91177139215794188-127.0.0.1:57543_kzqs%2Fqf-n_0000000046, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10394, name=Overseer-91177139215794188-127.0.0.1:57543_kzqs%2Fqf-n_0000000046, state=RUNNABLE, group=Overseer collection creation process.]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([737BFC392854E4BF]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:202)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:290)
	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:287)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:287)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:473)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:190)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:193)
	... 1 more




Build Log:
[...truncated 10754 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 3096530 T10175 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /kzqs/qf
   [junit4]   2> 3096535 T10175 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391252734610
   [junit4]   2> 3096536 T10175 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 3096537 T10176 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3096639 T10175 oasc.ZkTestServer.run start zk server on port:57527
   [junit4]   2> 3096641 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3096648 T10182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@762ba316 name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3096649 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3096649 T10175 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3096659 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3096663 T10184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ebf805e name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3096664 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3096664 T10175 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3096671 T10175 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3096678 T10175 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3096684 T10175 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3096692 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3096692 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3096703 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3096704 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3096713 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3096713 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3096720 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3096721 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3096728 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3096729 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3096736 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3096737 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3096744 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3096745 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3096750 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3096751 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3096758 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3096758 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3096765 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3096765 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3096773 T10175 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3096774 T10175 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3097151 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3097161 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57530
   [junit4]   2> 3097163 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3097163 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3097164 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858
   [junit4]   2> 3097165 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/'
   [junit4]   2> 3097213 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/solr.xml
   [junit4]   2> 3097332 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3097333 T10175 oasc.CoreContainer.<init> New CoreContainer 666520580
   [junit4]   2> 3097333 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/]
   [junit4]   2> 3097335 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3097335 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3097336 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3097336 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3097336 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3097336 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3097337 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3097337 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3097337 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3097356 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3097357 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3097358 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3097358 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3097359 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3097361 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3097367 T10195 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d4aefc name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3097368 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3097375 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3097381 T10197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@347f4676 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3097382 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3097388 T10175 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3097401 T10175 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3097413 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3097421 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57530_kzqs%2Fqf
   [junit4]   2> 3097425 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57530_kzqs%2Fqf
   [junit4]   2> 3097438 T10175 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3097446 T10175 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3097460 T10175 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57530_kzqs%2Fqf
   [junit4]   2> 3097460 T10175 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3097468 T10175 oasc.Overseer.start Overseer (id=91177139215794179-127.0.0.1:57530_kzqs%2Fqf-n_0000000000) starting
   [junit4]   2> 3097478 T10175 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3097491 T10199 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3097492 T10175 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3097493 T10199 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 3097498 T10175 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3097504 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3097516 T10198 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3097519 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858
   [junit4]   2> 3097628 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/conf/
   [junit4]   2> 3097634 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3097635 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3097635 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3097637 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3097641 T10202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51bcae1e name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3097643 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3097646 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3097654 T10175 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3097991 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 3097992 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3097996 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57534
   [junit4]   2> 3097996 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3097997 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3097997 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728
   [junit4]   2> 3097997 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/'
   [junit4]   2> 3098033 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/solr.xml
   [junit4]   2> 3098124 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3098125 T10175 oasc.CoreContainer.<init> New CoreContainer 494009639
   [junit4]   2> 3098125 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/]
   [junit4]   2> 3098127 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3098127 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3098128 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3098128 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3098128 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3098129 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3098129 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3098129 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3098130 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3098140 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3098141 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3098141 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3098142 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3098142 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3098143 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3098148 T10213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26a31a8b name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3098148 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3098154 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3098158 T10215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c54f191 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3098159 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3098170 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3099180 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57534_kzqs%2Fqf
   [junit4]   2> 3099183 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57534_kzqs%2Fqf
   [junit4]   2> 3099191 T10202 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3099191 T10215 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3099191 T10197 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3099204 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728
   [junit4]   2> 3099363 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/conf/
   [junit4]   2> 3099371 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3099372 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3099372 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3099712 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 3099713 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3099716 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57537
   [junit4]   2> 3099717 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3099717 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3099718 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447
   [junit4]   2> 3099718 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/'
   [junit4]   2> 3099751 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/solr.xml
   [junit4]   2> 3099822 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3099823 T10175 oasc.CoreContainer.<init> New CoreContainer 2079092943
   [junit4]   2> 3099823 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/]
   [junit4]   2> 3099825 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3099825 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3099826 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3099826 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3099826 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3099827 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3099827 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3099827 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3099828 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3099838 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3099838 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3099839 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3099839 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3099839 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3099841 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3099845 T10227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fe26598 name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3099846 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3099852 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3099855 T10229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a42d398 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3099856 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3099866 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3100875 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57537_kzqs%2Fqf
   [junit4]   2> 3100877 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57537_kzqs%2Fqf
   [junit4]   2> 3100886 T10215 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3100886 T10197 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3100887 T10229 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3100887 T10202 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3100900 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447
   [junit4]   2> 3101067 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/conf/
   [junit4]   2> 3101075 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3101076 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3101076 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3101433 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 3101434 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3101438 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57540
   [junit4]   2> 3101439 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3101439 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3101440 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151
   [junit4]   2> 3101441 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/'
   [junit4]   2> 3101488 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/solr.xml
   [junit4]   2> 3101583 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3101583 T10175 oasc.CoreContainer.<init> New CoreContainer 1433565453
   [junit4]   2> 3101583 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/]
   [junit4]   2> 3101585 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3101585 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3101585 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3101586 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3101586 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3101586 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3101587 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3101587 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3101587 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3101599 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3101599 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3101600 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3101600 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3101600 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3101602 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3101608 T10241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e69704e name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3101609 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3101618 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3101622 T10243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6091519a name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3101622 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3101636 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3102647 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57540_kzqs%2Fqf
   [junit4]   2> 3102650 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57540_kzqs%2Fqf
   [junit4]   2> 3102658 T10243 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 3102659 T10202 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 3102659 T10229 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 3102660 T10197 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 3102660 T10215 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 3102677 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151
   [junit4]   2> 3102845 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/conf/
   [junit4]   2> 3102854 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3102855 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3102855 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3103245 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 3103246 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3103253 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57543
   [junit4]   2> 3103256 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3103258 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3103259 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930
   [junit4]   2> 3103260 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/'
   [junit4]   2> 3103298 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/solr.xml
   [junit4]   2> 3103369 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3103369 T10175 oasc.CoreContainer.<init> New CoreContainer 410991380
   [junit4]   2> 3103370 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/]
   [junit4]   2> 3103371 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3103371 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3103372 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3103372 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3103372 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3103373 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3103373 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3103373 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3103374 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3103384 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3103384 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3103384 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3103385 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3103385 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3103386 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3103399 T10255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48ffaa9c name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3103400 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3103405 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3103410 T10257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21c52883 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3103411 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3103423 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3104434 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57543_kzqs%2Fqf
   [junit4]   2> 3104437 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57543_kzqs%2Fqf
   [junit4]   2> 3104445 T10215 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104445 T10202 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104446 T10229 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104445 T10257 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104447 T10243 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104447 T10197 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 3104461 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930
   [junit4]   2> 3104634 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/conf/
   [junit4]   2> 3104642 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3104643 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3104643 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3105040 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 3105041 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3105046 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57546
   [junit4]   2> 3105047 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3105047 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3105048 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718
   [junit4]   2> 3105048 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/'
   [junit4]   2> 3105080 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/solr.xml
   [junit4]   2> 3105169 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3105170 T10175 oasc.CoreContainer.<init> New CoreContainer 1370491970
   [junit4]   2> 3105170 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/]
   [junit4]   2> 3105172 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3105172 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3105172 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3105173 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3105173 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3105173 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3105174 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3105174 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3105174 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3105185 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3105185 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3105185 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3105186 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3105186 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3105188 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3105193 T10269 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cf4eaf3 name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3105194 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3105199 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3105204 T10271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3976cde3 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3105204 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3105217 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3106228 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57546_kzqs%2Fqf
   [junit4]   2> 3106231 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57546_kzqs%2Fqf
   [junit4]   2> 3106238 T10271 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106239 T10243 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106239 T10229 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106239 T10215 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106240 T10197 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106239 T10202 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106240 T10257 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 3106252 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718
   [junit4]   2> 3106372 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/conf/
   [junit4]   2> 3106377 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3106378 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3106378 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3106764 T10175 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 3106765 T10175 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3106770 T10175 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57549
   [junit4]   2> 3106771 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3106771 T10175 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3106772 T10175 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453
   [junit4]   2> 3106772 T10175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/'
   [junit4]   2> 3106803 T10175 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/solr.xml
   [junit4]   2> 3106880 T10175 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 3106881 T10175 oasc.CoreContainer.<init> New CoreContainer 250394993
   [junit4]   2> 3106881 T10175 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/]
   [junit4]   2> 3106883 T10175 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3106883 T10175 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 3106883 T10175 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3106884 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3106884 T10175 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3106884 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3106884 T10175 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3106885 T10175 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3106885 T10175 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3106896 T10175 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3106897 T10175 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3106897 T10175 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3106897 T10175 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57527/solr
   [junit4]   2> 3106898 T10175 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3106899 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106904 T10283 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5782b15e name:ZooKeeperConnection Watcher:127.0.0.1:57527 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106905 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106911 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106916 T10285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6466af15 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106916 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106929 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3107943 T10175 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57549_kzqs%2Fqf
   [junit4]   2> 3107946 T10175 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57549_kzqs%2Fqf
   [junit4]   2> 3107957 T10243 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107958 T10215 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107958 T10202 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107959 T10257 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107959 T10285 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107958 T10197 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107959 T10229 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107959 T10271 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 3107978 T10175 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453
   [junit4]   2> 3108225 T10175 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/conf/
   [junit4]   2> 3108237 T10175 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3108238 T10175 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3108239 T10175 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3108241 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108246 T10288 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@408643a1 name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108247 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108250 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3108260 T10175 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108266 T10290 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4174897b name:ZooKeeperConnection Watcher:127.0.0.1:57527/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108267 T10175 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108271 T10175 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3108283 T10253 oasha.CollectionsHandler.handleCreateAction Creating Collection : name=testOverseerCol&version=2&numShards=4&wt=javabin&maxShardsPerNode=2&replicationFactor=2&action=CREATE
   [junit4]   2> 3108286 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108289 T10199 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 3108290 T10199 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 3108292 T10199 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 3108292 T10199 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 3108292 T10199 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 3108305 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108310 T10198 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3108313 T10198 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 3108313 T10198 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 3108320 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108324 T10229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108324 T10243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108324 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108325 T10197 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108325 T10271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108326 T10285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108324 T10257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108324 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108326 T10290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108325 T10288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108406 T10199 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 3108407 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:57537_kzqs%2Fqf
   [junit4]   2> 3108407 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:57549_kzqs%2Fqf
   [junit4]   2> 3108408 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:57540_kzqs%2Fqf
   [junit4]   2> 3108410 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:57534_kzqs%2Fqf
   [junit4]   2> 3108412 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:57546_kzqs%2Fqf
   [junit4]   2> 3108413 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:57530_kzqs%2Fqf
   [junit4]   2> 3108416 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:57543_kzqs%2Fqf
   [junit4]   2> 3108422 T10207 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 3108422 T10222 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 3108423 T10199 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:57537_kzqs%2Fqf
   [junit4]   2> 3108424 T10191 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 3108424 T10264 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 3108428 T10280 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 3108429 T10207 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108431 T10223 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 3108430 T10248 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 3108432 T10222 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108429 T10239 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 3108433 T10264 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108434 T10280 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108435 T10248 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108435 T10239 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108435 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108435 T10223 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108435 T10191 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108439 T10198 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3108441 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57534/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57534_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108448 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108450 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57537/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57537_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108457 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108459 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57530/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57530_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108465 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108468 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57546/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57546_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108475 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108477 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57549/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57549_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108484 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108487 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57543/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57543_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108492 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108495 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57537/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57537_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108502 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108505 T10198 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57540/kzqs/qf",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57540_kzqs%2Fqf",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3108512 T10197 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3108621 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108621 T10271 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10290 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108623 T10288 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3108622 T10197 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 3109432 T10207 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 3109434 T10222 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 3109436 T10207 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/testOverseerCol_shard2_replica2
   [junit4]   2> 3109436 T10207 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109437 T10264 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 3109437 T10222 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/testOverseerCol_shard1_replica1
   [junit4]   2> 3109438 T10222 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109439 T10280 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 3109439 T10264 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1
   [junit4]   2> 3109440 T10264 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109440 T10248 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 3109441 T10280 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2
   [junit4]   2> 3109442 T10280 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109442 T10239 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 3109443 T10248 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/testOverseerCol_shard4_replica1
   [junit4]   2> 3109443 T10248 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109444 T10223 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 3109446 T10239 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/testOverseerCol_shard2_replica1
   [junit4]   2> 3109447 T10239 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109447 T10191 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 3109448 T10223 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/testOverseerCol_shard4_replica2
   [junit4]   2> 3109448 T10223 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109451 T10191 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/testOverseerCol_shard3_replica2
   [junit4]   2> 3109451 T10191 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 3109453 T10248 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109453 T10248 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109454 T10280 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109454 T10280 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109455 T10222 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109455 T10222 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109454 T10207 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109456 T10207 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109456 T10239 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109458 T10239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109456 T10223 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109459 T10223 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109455 T10264 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109461 T10264 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109459 T10191 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3109462 T10191 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 3109463 T10280 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/'
   [junit4]   2> 3109463 T10248 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391252740930/testOverseerCol_shard4_replica1/'
   [junit4]   2> 3109469 T10207 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391252735728/testOverseerCol_shard2_replica2/'
   [junit4]   2> 3109470 T10239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391252739151/testOverseerCol_shard2_replica1/'
   [junit4]   2> 3109473 T10264 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/'
   [junit4]   2> 3109484 T10222 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/testOverseerCol_shard1_replica1/'
   [junit4]   2> 3109489 T10223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391252737447/testOverseerCol_shard4_replica2/'
   [junit4]   2> 3109498 T10191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391252734858/testOverseerCol_shard3_replica2/'
   [junit4]   2> 3109752 T10264 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109767 T10191 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109828 T10280 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109869 T10223 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109874 T10222 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109968 T10239 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3109992 T10264 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110023 T10264 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110031 T10248 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3110032 T10207 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3110064 T10264 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 3110149 T10280 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110174 T10280 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110220 T10222 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110229 T10280 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 3110238 T10191 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110272 T10222 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110282 T10222 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 3110285 T10191 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110289 T10239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110292 T10239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110303 T10239 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 3110310 T10223 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110317 T10223 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110329 T10223 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 3110337 T10191 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 3110366 T10207 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110376 T10207 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110385 T10207 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 3110387 T10248 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3110391 T10248 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3110429 T10248 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 3113249 T10264 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113265 T10280 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113320 T10280 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113323 T10280 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113327 T10264 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113330 T10264 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113405 T10191 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113433 T10191 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113437 T10191 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113494 T10248 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113503 T10264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3113517 T10280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3113518 T10248 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113521 T10248 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113523 T10264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3113532 T10280 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3113536 T10207 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113540 T10264 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3113545 T10207 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113550 T10207 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113550 T10223 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113567 T10223 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113568 T10264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3113571 T10264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3113572 T10264 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3113573 T10223 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113588 T10239 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3113589 T10280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3113592 T10264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3113593 T10264 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3113594 T10264 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3113594 T10280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3113595 T10280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3113606 T10280 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3113595 T10264 oasc.SolrCore.<init> [testOverseerCol_shard3_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/data/
   [junit4]   2> 3113607 T10264 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5791fc25
   [junit4]   2> 3113610 T10191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3113611 T10280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 3113612 T10280 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 3113612 T10280 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3113613 T10280 oasc.SolrCore.<init> [testOverseerCol_shard1_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/data/
   [junit4]   2> 3113613 T10280 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5791fc25
   [junit4]   2> 3113614 T10280 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/data
   [junit4]   2> 3113614 T10280 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/data/index/
   [junit4]   2> 3113615 T10280 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3113601 T10239 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3113610 T10264 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/data
   [junit4]   2> 3113618 T10264 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/data/index/
   [junit4]   2> 3113618 T10264 oasc.SolrCore.initIndex WARN [testOverseerCol_shard3_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391252742718/testOverseerCol_shard3_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3113616 T10280 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391252744453/testOverseerCol_shard1_replica2/data/index
   [junit4]   2> 3113620 T10280 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=45.6748046875, floorSegmentMB=1.3388671875, forceMergeDeletesPctAllowed=29.660997853766712, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.609434232716509
   [junit4]   2> 3113627 T10239 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3113630 T10264 oasc.CachingDirectoryFa

[...truncated too long message...]

StateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3122628 T10243 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3122729 T10257 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3122729 T10257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3122730 T10257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3122730 T10257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3122730 T10257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3122823 T10271 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3122937 T10285 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=nl_NL, timezone=Pacific/Kosrae
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=6,free=99414624,total=284327936
   [junit4]   2> NOTE: All tests run in this JVM: [TestBinaryResponseWriter, TestImplicitCoreProperties, QueryParsingTest, TermVectorComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestFieldTypeResource, SliceStateUpdateTest, BasicZkTest, TestPartialUpdateDeduplication, ExpressionTest, UnloadDistributedZkTest, TestZkChroot, CoreAdminHandlerTest, TestSolrXmlPersistor, TestSerializedLuceneMatchVersion, TestRandomFaceting, TestRealTimeGet, TestPerFieldSimilarity, SpellingQueryConverterTest, DistributedTermsComponentTest, HdfsUnloadDistributedZkTest, TestStressUserVersions, TestFunctionQuery, MBeansHandlerTest, FieldAnalysisRequestHandlerTest, CachingDirectoryFactoryTest, CoreContainerCoreInitFailuresTest, TestPostingsSolrHighlighter, BasicDistributedZkTest, AutoCommitTest, DisMaxRequestHandlerTest, CurrencyFieldXmlFileTest, TestCSVResponseWriter, TestMaxScoreQueryParser, SolrXmlInZkTest, TestBM25SimilarityFactory, XmlUpdateRequestHandlerTest, TestStressLucene, ConvertedLegacyTest, XsltUpdateRequestHandlerTest, LegacyHTMLStripCharFilterTest, ZkCLITest, TestWriterPerf, PingRequestHandlerTest, LoggingHandlerTest, FieldMutatingUpdateProcessorTest, ParsingFieldUpdateProcessorsTest, DistributedSuggestComponentTest, TestReplicationHandler, DocValuesTest, TestFieldTypeCollectionResource, LeaderElectionIntegrationTest, TestCollationFieldDocValues, CursorMarkTest, TestFileDictionaryLookup, CSVRequestHandlerTest, StandardRequestHandlerTest, TestManagedSchema, TestCollapseQParserPlugin, TestValueSourceCache, TestSolrCoreProperties, StressHdfsTest, TestLMDirichletSimilarityFactory, DistribCursorPagingTest, ResponseLogComponentTest, UniqFieldsUpdateProcessorFactoryTest, OpenCloseCoreStressTest, TestMultiCoreConfBootstrap, ResourceLoaderTest, TestStressRecovery, DistributedDebugComponentTest, PrimUtilsTest, DOMUtilTest, SolrCmdDistributorTest, MultiTermTest, AlternateDirectoryTest, TestLFUCache, DirectSolrSpellCheckerTest, TestStressReorder, TestSchemaVersionResource, AbstractAnalyticsFacetTest, HdfsCollectionsAPIDistributedZkTest, TestCollationField, DirectUpdateHandlerOptimizeTest, NoCacheHeaderTest, TestQuerySenderListener, DistributedQueryComponentCustomSortTest, ReturnFieldsTest, ShardRoutingTest, AnalysisAfterCoreReloadTest, TestCloudManagedSchemaAddField, TestRandomMergePolicy, TestModifyConfFiles, DeleteShardTest, ChaosMonkeySafeLeaderTest, TestDefaultSearchFieldResource, BadIndexSchemaTest, TestDocBasedVersionConstraints, SpellCheckCollatorTest, TestBadConfig, UpdateRequestProcessorFactoryTest, TestOmitPositions, HdfsBasicDistributedZkTest, TestHighFrequencyDictionaryFactory, TestIndexingPerformance, TriLevelCompositeIdRoutingTest, IndexSchemaTest, SystemInfoHandlerTest, HdfsLockFactoryTest, SpatialFilterTest, TestQuerySenderNoQuery, BadComponentTest, TestSolr4Spatial, IndexBasedSpellCheckerTest, QueryElevationComponentTest, MinimalSchemaTest, DistanceFunctionTest, TestDFRSimilarityFactory, QueryResultKeyTest, TestRemoteStreaming, DistributedSpellCheckComponentTest, SoftAutoCommitTest, TestLRUCache, BinaryUpdateRequestHandlerTest, SampleTest, InfoHandlerTest, SearchHandlerTest, TestUniqueKeyFieldResource, TestFieldCollectionResource, TestQueryUtils, TestSolrQueryParser, PeerSyncTest, BasicFunctionalityTest, DocValuesMissingTest, TestSolrDeletionPolicy1, TestPseudoReturnFields, MoreLikeThisHandlerTest, DeleteReplicaTest, TestExtendedDismaxParser, TestGroupingSearch, QueryEqualityTest, TestSuggestSpellingConverter, StatsComponentTest, TestSort, TestCustomSort, TestAnalyzeInfixSuggestions, TestXIncludeConfig, TestUpdate, TestNumberUtils, RequiredFieldsTest, EchoParamsTest, TestElisionMultitermQuery, UUIDFieldTest, SynonymTokenizerTest, CopyFieldTest, LukeRequestHandlerTest, IndexSchemaRuntimeFieldTest, TestSystemIdResolver, TestCloudManagedSchema, ConnectionManagerTest, TestNoOpRegenerator, TestFuzzyAnalyzedSuggestions, ZkControllerTest, SchemaVersionSpecificBehaviorTest, SpellCheckComponentTest, DocValuesMultiTest, TestSolrXMLSerializer, TestLMJelinekMercerSimilarityFactory, CoreAdminCreateDiscoverTest, BJQParserTest, TestRecovery, ShardRoutingCustomTest, MigrateRouteKeyTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, OverseerTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, TestRandomDVFaceting, TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestJoin, TestReload, TestStressVersions, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, SolrCoreTest, TestFiltering, TestLazyCores, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterTest, TestTrie, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, SortByFunctionTest, SolrInfoMBeanTest, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, TermsComponentTest, RequestHandlersTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestArbitraryIndexDir, FastVectorHighlighterTest, RegexBoostProcessorTest, SolrPluginUtilsTest, TestJmxIntegration, JSONWriterTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPHPSerializedResponseWriter, TestComponentsName, HighlighterConfigTest, SOLR749Test, UpdateParamsTest, TestSolrIndexConfig, SolrIndexConfigTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, TestSweetSpotSimilarityFactory, TestIBSimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TimeZoneUtilsTest, TestFastOutputStream, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, DateFieldTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, FileUtilsTest, CircularListTest, TestRTGBase, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, AbstractAnalyticsStatsTest, NoFacetTest, FieldFacetExtrasTest, QueryFacetTest, RangeFacetTest, FunctionTest, AliasIntegrationTest, AssignTest, CustomCollectionTest, OverseerRolesTest]
   [junit4] Completed in 26.44s, 1 test, 1 error <<< FAILURES!

[...truncated 136 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 error, 40 ignored (6 assumptions)

Total time: 89 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1285 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1285/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 10350 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140201_232742_828.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] java(211,0x145004000) malloc: *** error for object 0x10000144ff3080: pointer being freed was not allocated
   [junit4] *** set a breakpoint in malloc_error_break to debug
   [junit4] <<< JVM J0: EOF ----

[...truncated 1 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/bin/java -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/heapdumps -Dtests.prefix=tests -Dtests.seed=237A3ACAC48BA2B -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=5.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=5.0-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.disableHdfs=true -Dfile.encoding=ISO-8859-1 -classpath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/test:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/test-framework/lib/junit4-ant-2.0.13.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-solrj/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/common/lucene-analyzers-common-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/lucene-codecs-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/highlighter/lucene-highlighter-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/memory/lucene-memory-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/misc/lucene-misc-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/spatial/lucene-spatial-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/expressions/lucene-expressions-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/suggest/lucene-suggest-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/grouping/lucene-grouping-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queries/lucene-queries-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queryparser/lucene-queryparser-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/join/lucene-join-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/antlr-runtime-3.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-commons-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-cli-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-codec-1.7.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-configuration-1.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-fileupload-1.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-lang-2.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/guava-14.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-annotations-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-auth-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-common-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-hdfs-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hppc-0.5.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/joda-time-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/protobuf-java-2.5.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/spatial4j-0.4.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/commons-io-2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpclient-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpcore-4.3.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpmime-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/jcl-over-slf4j-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/jul-to-slf4j-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/log4j-1.2.16.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/noggit-0.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/slf4j-api-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/slf4j-log4j12-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/wstx-asl-3.2.7.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/zookeeper-3.4.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-continuation-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-deploy-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-http-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-io-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-jmx-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-security-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-server-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-servlet-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-util-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-webapp-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-xml-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/servlet-api-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/junit-4.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/randomizedtesting-runner-2.0.13.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/antlr-runtime-3.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/asm-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/asm-commons-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/cglib-nodep-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/commons-collections-3.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/easymock-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hppc-0.5.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jersey-core-1.8.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-util-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/objenesis-1.2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/Users/jenkins/.ant/lib/ivy-2.3.0.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/lib/tools.jar:/Users/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.13.jar -ea:org.apache.lucene... -ea:org.apache.solr... com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -flush -eventsfile /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140201_232742_828.events @/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140201_232742_828.suites
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1373)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:119)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:911)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:908)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:744)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.

Total time: 68 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1284 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1284/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=3473, name=Overseer-91178394928414736-127.0.0.1:50320_ie_uaf%2Ft-n_0000000016, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3473, name=Overseer-91178394928414736-127.0.0.1:50320_ie_uaf%2Ft-n_0000000016, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([BABFEB6317D026A2:3B59657B608F469E]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([BABFEB6317D026A2]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:202)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:193)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:193)
	... 1 more




Build Log:
[...truncated 10048 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 916436 T3306 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ie_uaf/t
   [junit4]   2> 916445 T3306 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391271895261
   [junit4]   2> 916447 T3306 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 916448 T3307 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 916549 T3306 oasc.ZkTestServer.run start zk server on port:50298
   [junit4]   2> 916551 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 916557 T3313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30d21291 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 916558 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 916558 T3306 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 916571 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 916576 T3315 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17398876 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 916577 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 916578 T3306 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 916588 T3306 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 916595 T3306 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 916601 T3306 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 916608 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 916609 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 916623 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 916624 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 916632 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 916632 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 916640 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 916641 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 916649 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 916650 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 916659 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 916660 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 916667 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 916669 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 916678 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 916679 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 916688 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 916688 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 916697 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 916698 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 916706 T3306 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 916707 T3306 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 917077 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 917084 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50301
   [junit4]   2> 917084 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 917085 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 917085 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534
   [junit4]   2> 917085 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/'
   [junit4]   2> 917120 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/solr.xml
   [junit4]   2> 917200 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 917201 T3306 oasc.CoreContainer.<init> New CoreContainer 1083546251
   [junit4]   2> 917201 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/]
   [junit4]   2> 917203 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 917203 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 917203 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 917204 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 917204 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 917205 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 917205 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 917205 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 917205 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 917215 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 917216 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 917216 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 917216 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 917217 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 917218 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 917225 T3326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dd53ccd name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 917226 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 917230 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 917240 T3328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@284cf666 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 917240 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 917243 T3306 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 917254 T3306 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 917261 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 917265 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50301_ie_uaf%2Ft
   [junit4]   2> 917266 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50301_ie_uaf%2Ft
   [junit4]   2> 917274 T3306 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 917282 T3306 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 917292 T3306 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50301_ie_uaf%2Ft
   [junit4]   2> 917293 T3306 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 917296 T3306 oasc.Overseer.start Overseer (id=91178394928414723-127.0.0.1:50301_ie_uaf%2Ft-n_0000000000) starting
   [junit4]   2> 917302 T3306 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 917309 T3306 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 917311 T3330 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 917313 T3330 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 917315 T3306 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 917320 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 917328 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534
   [junit4]   2> 917367 T3329 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 917441 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/conf/
   [junit4]   2> 917446 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 917447 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 917447 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 917449 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 917453 T3333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ac98b56 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 917454 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 917459 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 917466 T3306 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 917832 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 917832 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 917837 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50305
   [junit4]   2> 917839 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 917839 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 917840 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281
   [junit4]   2> 917841 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/'
   [junit4]   2> 917871 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/solr.xml
   [junit4]   2> 917960 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 917962 T3306 oasc.CoreContainer.<init> New CoreContainer 1876494494
   [junit4]   2> 917962 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/]
   [junit4]   2> 917964 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 917964 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 917964 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 917964 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 917965 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 917965 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 917965 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 917965 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 917966 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 917975 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 917976 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 917976 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 917976 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 917977 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 917978 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 917982 T3344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c4f670 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 917983 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 917995 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 918001 T3346 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c73c1c0 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 918002 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 918014 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 919027 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50305_ie_uaf%2Ft
   [junit4]   2> 919029 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50305_ie_uaf%2Ft
   [junit4]   2> 919036 T3346 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 919037 T3328 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 919037 T3333 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 919051 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281
   [junit4]   2> 919192 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/conf/
   [junit4]   2> 919199 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 919200 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 919200 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 919551 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 919552 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 919557 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50308
   [junit4]   2> 919559 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 919559 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 919559 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016
   [junit4]   2> 919560 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/'
   [junit4]   2> 919591 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/solr.xml
   [junit4]   2> 919666 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 919666 T3306 oasc.CoreContainer.<init> New CoreContainer 1063807477
   [junit4]   2> 919667 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/]
   [junit4]   2> 919668 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 919668 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 919668 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 919669 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 919669 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 919669 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 919669 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 919670 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 919670 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 919680 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 919680 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 919681 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 919681 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 919681 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 919683 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 919687 T3358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60047458 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 919689 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 919696 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 919701 T3360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@241bfc05 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 919702 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 919722 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 920735 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50308_ie_uaf%2Ft
   [junit4]   2> 920737 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50308_ie_uaf%2Ft
   [junit4]   2> 920746 T3333 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 920746 T3346 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 920746 T3328 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 920746 T3360 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 920757 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016
   [junit4]   2> 920854 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/conf/
   [junit4]   2> 920859 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 920860 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 920860 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 921212 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 921214 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 921218 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50311
   [junit4]   2> 921221 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 921222 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 921222 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676
   [junit4]   2> 921223 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/'
   [junit4]   2> 921249 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/solr.xml
   [junit4]   2> 921316 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 921316 T3306 oasc.CoreContainer.<init> New CoreContainer 511120476
   [junit4]   2> 921316 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/]
   [junit4]   2> 921318 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 921318 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 921318 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 921319 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 921319 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 921319 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 921319 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 921320 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 921321 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 921331 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 921331 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 921332 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 921332 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 921332 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 921334 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 921338 T3372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd9b3a4 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 921341 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 921348 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 921352 T3374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab24ac3 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 921352 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 921365 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 922380 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50311_ie_uaf%2Ft
   [junit4]   2> 922382 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50311_ie_uaf%2Ft
   [junit4]   2> 922390 T3328 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 922390 T3346 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 922391 T3360 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 922391 T3333 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 922391 T3374 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 922405 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676
   [junit4]   2> 922562 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/conf/
   [junit4]   2> 922573 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 922575 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 922575 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 922931 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 922931 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 922937 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50314
   [junit4]   2> 922939 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 922939 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 922939 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391
   [junit4]   2> 922940 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/'
   [junit4]   2> 922969 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/solr.xml
   [junit4]   2> 923039 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 923040 T3306 oasc.CoreContainer.<init> New CoreContainer 1603034028
   [junit4]   2> 923041 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/]
   [junit4]   2> 923042 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 923042 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 923042 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 923043 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 923043 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 923043 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 923044 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 923044 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 923045 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 923055 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 923055 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 923056 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 923056 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 923057 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 923058 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 923063 T3386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77d6aa35 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 923064 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 923075 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 923083 T3388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76252f01 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 923083 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 923090 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 924098 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50314_ie_uaf%2Ft
   [junit4]   2> 924104 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50314_ie_uaf%2Ft
   [junit4]   2> 924119 T3346 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924129 T3328 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924129 T3360 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924134 T3374 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924134 T3333 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924135 T3388 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 924158 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391
   [junit4]   2> 924277 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/conf/
   [junit4]   2> 924284 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 924286 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 924287 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 924699 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 924700 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 924706 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50317
   [junit4]   2> 924707 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 924707 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 924707 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103
   [junit4]   2> 924708 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/'
   [junit4]   2> 924746 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/solr.xml
   [junit4]   2> 924830 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 924831 T3306 oasc.CoreContainer.<init> New CoreContainer 1580173652
   [junit4]   2> 924832 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/]
   [junit4]   2> 924834 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 924834 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 924834 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 924835 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 924835 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 924835 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 924836 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 924836 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 924837 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 924849 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 924849 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 924850 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 924850 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 924850 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 924852 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 924856 T3400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e11a1b name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 924857 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 924863 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 924867 T3402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b1a3ce6 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 924868 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 924878 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 925889 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50317_ie_uaf%2Ft
   [junit4]   2> 925895 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50317_ie_uaf%2Ft
   [junit4]   2> 925915 T3374 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925916 T3388 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925917 T3333 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925917 T3360 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925917 T3402 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925919 T3346 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925919 T3328 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 925943 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103
   [junit4]   2> 926109 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/conf/
   [junit4]   2> 926117 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 926118 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 926119 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 926462 T3306 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 926463 T3306 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 926467 T3306 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50320
   [junit4]   2> 926469 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 926469 T3306 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 926470 T3306 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934
   [junit4]   2> 926470 T3306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/'
   [junit4]   2> 926506 T3306 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/solr.xml
   [junit4]   2> 926586 T3306 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 926586 T3306 oasc.CoreContainer.<init> New CoreContainer 226246337
   [junit4]   2> 926587 T3306 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/]
   [junit4]   2> 926588 T3306 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 926589 T3306 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 926590 T3306 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 926590 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 926590 T3306 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 926591 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 926591 T3306 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 926592 T3306 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 926592 T3306 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 926602 T3306 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 926603 T3306 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 926603 T3306 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 926604 T3306 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50298/solr
   [junit4]   2> 926604 T3306 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 926606 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 926610 T3414 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57fbf7d1 name:ZooKeeperConnection Watcher:127.0.0.1:50298 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 926611 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 926618 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 926623 T3416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21c9cc0c name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 926624 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 926634 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 927647 T3306 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50320_ie_uaf%2Ft
   [junit4]   2> 927653 T3306 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50320_ie_uaf%2Ft
   [junit4]   2> 927669 T3333 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927670 T3374 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927670 T3328 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927672 T3360 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927675 T3402 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927676 T3416 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927676 T3346 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927676 T3388 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 927691 T3306 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934
   [junit4]   2> 927785 T3306 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/conf/
   [junit4]   2> 927792 T3306 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 927793 T3306 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 927793 T3306 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 927796 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 927801 T3419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e926d63 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 927801 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 927804 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 927813 T3306 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 927816 T3421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3420158 name:ZooKeeperConnection Watcher:127.0.0.1:50298/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 927817 T3306 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 927820 T3306 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 927829 T3337 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&name=testOverseerCol&numShards=4&replicationFactor=2&action=CREATE&maxShardsPerNode=2&wt=javabin
   [junit4]   2> 927832 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 927836 T3330 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 927836 T3330 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 927838 T3330 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 927838 T3330 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 927838 T3330 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 927853 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 927858 T3329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 927860 T3329 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 927860 T3329 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 927872 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 927876 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3402 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3419 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3346 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3416 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927877 T3421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 927954 T3330 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 927955 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:50314_ie_uaf%2Ft
   [junit4]   2> 927956 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:50317_ie_uaf%2Ft
   [junit4]   2> 927959 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:50308_ie_uaf%2Ft
   [junit4]   2> 927960 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:50311_ie_uaf%2Ft
   [junit4]   2> 927961 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:50320_ie_uaf%2Ft
   [junit4]   2> 927966 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:50301_ie_uaf%2Ft
   [junit4]   2> 927970 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:50305_ie_uaf%2Ft
   [junit4]   2> 927980 T3384 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 927982 T3330 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:50314_ie_uaf%2Ft
   [junit4]   2> 927986 T3395 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 927984 T3410 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 927986 T3338 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 927987 T3369 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 927987 T3353 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 927991 T3384 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927992 T3322 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 927992 T3410 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927992 T3353 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927993 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 927995 T3395 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927995 T3338 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927996 T3379 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 927996 T3369 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927998 T3322 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 927999 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 927999 T3379 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 928002 T3329 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 928003 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50314/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50314_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928013 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50308/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50308_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928020 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928024 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50320/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50320_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928033 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928036 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50317/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50317_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928044 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928048 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50305/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50305_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928055 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928058 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50311/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50311_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928065 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928068 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50301/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50301_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928078 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928081 T3329 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50314/ie_uaf/t",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50314_ie_uaf%2Ft",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 928090 T3328 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 928202 T3360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928203 T3388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928204 T3421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928204 T3419 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928204 T3402 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928206 T3346 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928204 T3328 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928205 T3416 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928205 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928204 T3333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 928992 T3384 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 928994 T3410 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 928997 T3395 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 928998 T3410 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/testOverseerCol_shard3_replica1
   [junit4]   2> 929000 T3410 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929000 T3353 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 929003 T3369 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 929004 T3353 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/testOverseerCol_shard2_replica1
   [junit4]   2> 929004 T3353 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929005 T3338 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 929007 T3369 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/testOverseerCol_shard2_replica2
   [junit4]   2> 929008 T3369 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929009 T3395 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2
   [junit4]   2> 929010 T3395 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929011 T3338 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1
   [junit4]   2> 929012 T3338 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929010 T3379 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 929010 T3322 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 929016 T3379 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard4_replica2
   [junit4]   2> 929016 T3379 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929011 T3384 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard1_replica1
   [junit4]   2> 929020 T3384 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929021 T3410 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929022 T3410 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929020 T3322 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/testOverseerCol_shard3_replica2
   [junit4]   2> 929023 T3322 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 929023 T3379 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929024 T3379 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929023 T3395 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929024 T3395 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929023 T3338 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929025 T3338 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929024 T3384 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929026 T3384 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929024 T3353 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929026 T3353 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929024 T3369 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929027 T3369 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929026 T3322 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 929028 T3322 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 929030 T3379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard4_replica2/'
   [junit4]   2> 929031 T3395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2/'
   [junit4]   2> 929031 T3410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1391271904934/testOverseerCol_shard3_replica1/'
   [junit4]   2> 929034 T3369 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391271899676/testOverseerCol_shard2_replica2/'
   [junit4]   2> 929036 T3338 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/'
   [junit4]   2> 929036 T3322 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391271895534/testOverseerCol_shard3_replica2/'
   [junit4]   2> 929037 T3384 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard1_replica1/'
   [junit4]   2> 929045 T3353 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391271898016/testOverseerCol_shard2_replica1/'
   [junit4]   2> 929265 T3395 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929309 T3379 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929309 T3322 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929321 T3384 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929335 T3369 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929355 T3353 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929368 T3410 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929431 T3338 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 929503 T3395 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929506 T3369 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929512 T3395 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929545 T3395 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 929578 T3384 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929585 T3384 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929590 T3369 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929643 T3379 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929648 T3322 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929653 T3379 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929670 T3322 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929729 T3369 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 929743 T3384 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 929762 T3379 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 929765 T3322 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 929799 T3353 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929820 T3353 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929838 T3353 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 929941 T3410 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 929944 T3410 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 929992 T3410 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 930006 T3338 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 930016 T3338 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 930032 T3338 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 932309 T3338 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932326 T3338 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932330 T3338 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932478 T3338 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932493 T3338 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932501 T3384 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932507 T3353 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932525 T3353 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932553 T3384 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932555 T3353 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932558 T3384 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932578 T3338 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 932580 T3322 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932593 T3338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 932593 T3338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 932594 T3338 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 932601 T3338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 932601 T3338 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 932602 T3338 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 932602 T3338 oasc.SolrCore.<init> [testOverseerCol_shard4_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/data/
   [junit4]   2> 932602 T3338 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c8633e1
   [junit4]   2> 932610 T3322 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932612 T3338 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/data
   [junit4]   2> 932622 T3395 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932623 T3338 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/data/index/
   [junit4]   2> 932624 T3338 oasc.SolrCore.initIndex WARN [testOverseerCol_shard4_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 932635 T3322 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932636 T3395 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932639 T3338 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391271896281/testOverseerCol_shard4_replica1/data/index
   [junit4]   2> 932641 T3338 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=54.408203125, floorSegmentMB=1.333984375, forceMergeDeletesPctAllowed=22.827411111393204, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.43762038431474526
   [junit4]   2> 932643 T3395 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932671 T3379 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932689 T3369 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932713 T3338 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@777c6d5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30fe711f),segFN=segments_1,generation=1}
   [junit4]   2> 932713 T3338 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 932725 T3384 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932726 T3338 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 932731 T3369 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932730 T3410 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 932728 T3379 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932736 T3379 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932738 T3410 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 932731 T3338 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 932742 T3338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 932742 T3338 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 932743 T3338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 932743 T3338 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 932743 T3338 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 932744 T3338 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 932744 T3338 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 932745 T3338 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 932745 T3338 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 932745 T3338 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 932747 T3338 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 932751 T3338 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 932751 T3338 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 932752 T3338 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 932752 T3338 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 932755 T3353 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932767 T3338 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 932779 T3384 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932783 T3353 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 932791 T3369 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932801 T3338 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 932802 T3338 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 932803 T3384 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 932818 T3410 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 932824 T3384 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 932825 T3384 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 932825 T3384 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 932831 T3353 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 932839 T3384 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 932839 T3384 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 932839 T3384 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 932840 T3384 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard1_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391271901391/testOverseerCol_shard1_replica1/data/
   [junit4]   2> 932841 T3384 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c8633e1
   [junit4]   2> 932831 T3338 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=30.4091796875, floorSegmentMB=1.

[...truncated too long message...]

st/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2/data/index
   [junit4]   2> 941833 T3402 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2/data [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2/data;done=false>>]
   [junit4]   2> 941833 T3402 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391271903103/testOverseerCol_shard1_replica2/data
   [junit4]   2> 941834 T3402 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leaders/shard1
   [junit4]   2> 941834 T3402 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 941836 T3402 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard1/election/91178394928414734-core_node4-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 941836 T3402 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 941837 T3402 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=en_MT, timezone=Australia/Queensland
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=2,free=150443448,total=339738624
   [junit4]   2> NOTE: All tests run in this JVM: [WordBreakSolrSpellCheckerTest, TestSchemaVersionResource, TestMultiCoreConfBootstrap, SuggesterWFSTTest, TestPHPSerializedResponseWriter, ConvertedLegacyTest, TestIndexSearcher, TestBM25SimilarityFactory, TestCSVResponseWriter, TestAnalyzeInfixSuggestions, TestCharFilters, PreAnalyzedFieldTest, DistributedTermsComponentTest, DeleteReplicaTest, TestRecoveryHdfs, SynonymTokenizerTest, TestRTGBase, HdfsRecoveryZkTest, IndexBasedSpellCheckerTest, TestExtendedDismaxParser, TestStressReorder, SolrCmdDistributorTest, TestPhraseSuggestions, TestDFRSimilarityFactory, TestRandomDVFaceting, TestUtils, TestIBSimilarityFactory, TestLRUCache, QueryFacetTest, StressHdfsTest, NotRequiredUniqueKeyTest, StandardRequestHandlerTest, NumericFieldsTest, ChaosMonkeyNothingIsSafeTest, SliceStateTest, TestRecovery, TestNonNRTOpen, RegexBoostProcessorTest, SuggesterTest, AnalysisAfterCoreReloadTest, SliceStateUpdateTest, QueryElevationComponentTest, TestSearchPerf, TestReload, AssignTest, TestCopyFieldCollectionResource, TestClassNameShortening, ExternalFileFieldSortTest, TestFastOutputStream, DocValuesTest, TestDynamicFieldResource, HdfsBasicDistributedZkTest, FunctionTest, TestSurroundQueryParser, TestUniqueKeyFieldResource, UpdateParamsTest, TestSystemIdResolver, TestQuerySenderListener, TestCSVLoader, TestCodecSupport, TestSweetSpotSimilarityFactory, TestSolrJ, TestValueSourceCache, ShardSplitTest, SuggestComponentTest, CursorPagingTest, TestFiltering, FieldFacetExtrasTest, TestSolrXmlPersistor, TriLevelCompositeIdRoutingTest, OutputWriterTest, SolrRequestParserTest, RAMDirectoryFactoryTest, StatelessScriptUpdateProcessorFactoryTest, SolrIndexSplitterTest, SoftAutoCommitTest, TestSimpleQParserPlugin, ChaosMonkeySafeLeaderTest, LukeRequestHandlerTest, RangeFacetTest, RequestHandlersTest, AddSchemaFieldsUpdateProcessorFactoryTest, ZkSolrClientTest, TestBinaryResponseWriter, TestWriterPerf, TestSolrXMLSerializer, TestConfig, TestRandomMergePolicy, TestJmxMonitoredMap, UpdateRequestProcessorFactoryTest, BJQParserTest, OverseerRolesTest]
   [junit4] Completed in 26.27s, 1 test, 1 error <<< FAILURES!

[...truncated 847 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 error, 40 ignored (6 assumptions)

Total time: 89 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure