You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/08 04:49:35 UTC

[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1276 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1276/
Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=10090, name=Overseer-91215041905360910-127.0.0.1:61003_-n_0000000012, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10090, name=Overseer-91215041905360910-127.0.0.1:61003_-n_0000000012, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([7B9CDA0FF7DAAF71:FA7A54178085CF4D]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([7B9CDA0FF7DAAF71]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:209)
	at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:193)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:200)
	... 1 more




Build Log:
[...truncated 10213 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 2818833 T9918 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2818837 T9918 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391831083755
   [junit4]   2> 2818838 T9918 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2818840 T9919 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2818941 T9918 oasc.ZkTestServer.run start zk server on port:60984
   [junit4]   2> 2818943 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2818953 T9925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d879d8 name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2818954 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2818955 T9918 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2818969 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2818974 T9927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59a2e549 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2818975 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2818976 T9918 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2818986 T9918 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2818995 T9918 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2819003 T9918 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2819012 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2819013 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2819027 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2819028 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2819071 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2819072 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2819103 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2819104 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2819111 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2819112 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2819119 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2819121 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2819128 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2819129 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2819138 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2819139 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2819149 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2819149 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2819159 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2819159 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2819184 T9918 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2819185 T9918 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2819524 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2819531 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60987
   [junit4]   2> 2819532 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2819532 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2819533 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117
   [junit4]   2> 2819533 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/'
   [junit4]   2> 2819559 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/solr.xml
   [junit4]   2> 2819599 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2819600 T9918 oasc.CoreContainer.<init> New CoreContainer 2049374808
   [junit4]   2> 2819600 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/]
   [junit4]   2> 2819602 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2819602 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2819602 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2819603 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2819603 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2819605 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2819605 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2819605 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2819606 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2819627 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2819627 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2819628 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2819628 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2819629 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2819630 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2819635 T9938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14fde60f name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2819636 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2819642 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2819646 T9940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24abf055 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2819647 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2819651 T9918 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2819664 T9918 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2819676 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2819683 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60987_
   [junit4]   2> 2819686 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60987_
   [junit4]   2> 2819697 T9918 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2819705 T9918 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2819720 T9918 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60987_
   [junit4]   2> 2819721 T9918 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2819729 T9918 oasc.Overseer.start Overseer (id=91215041905360899-127.0.0.1:60987_-n_0000000000) starting
   [junit4]   2> 2819740 T9918 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2819754 T9942 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2819756 T9918 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2819757 T9942 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2819764 T9918 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2819771 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2819782 T9941 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2819785 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117
   [junit4]   2> 2819932 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/conf/
   [junit4]   2> 2819938 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2819940 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2819941 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2819942 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2819947 T9945 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62cfb594 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2819948 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2819952 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2819961 T9918 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2820296 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2820297 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2820302 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60991
   [junit4]   2> 2820303 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2820303 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2820304 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879
   [junit4]   2> 2820304 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/'
   [junit4]   2> 2820332 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/solr.xml
   [junit4]   2> 2820406 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2820408 T9918 oasc.CoreContainer.<init> New CoreContainer 521848872
   [junit4]   2> 2820409 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/]
   [junit4]   2> 2820410 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2820411 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2820411 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2820412 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2820413 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2820413 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2820414 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2820414 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2820415 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2820436 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2820438 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2820438 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2820439 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2820439 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2820441 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2820448 T9956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@174aca0c name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2820449 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2820457 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2820462 T9958 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dad147 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2820463 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2820478 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2821493 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60991_
   [junit4]   2> 2821496 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60991_
   [junit4]   2> 2821507 T9940 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821508 T9945 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821510 T9958 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2821531 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879
   [junit4]   2> 2821641 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/conf/
   [junit4]   2> 2821646 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2821647 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2821648 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2821969 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2821970 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2821975 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60994
   [junit4]   2> 2821976 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2821976 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2821977 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565
   [junit4]   2> 2821978 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/'
   [junit4]   2> 2822007 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/solr.xml
   [junit4]   2> 2822051 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2822053 T9918 oasc.CoreContainer.<init> New CoreContainer 1412536956
   [junit4]   2> 2822053 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/]
   [junit4]   2> 2822054 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2822055 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2822055 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2822056 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2822057 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2822057 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2822057 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2822058 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2822059 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2822086 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2822087 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2822088 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2822089 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2822089 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2822093 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2822099 T9970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4956e13 name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2822100 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2822108 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2822112 T9972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@293d7b81 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2822114 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2822128 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2823142 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60994_
   [junit4]   2> 2823146 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60994_
   [junit4]   2> 2823157 T9940 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2823158 T9958 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2823160 T9945 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2823161 T9972 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2823178 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565
   [junit4]   2> 2823338 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/conf/
   [junit4]   2> 2823346 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2823347 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2823347 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2823724 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2823726 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2823731 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60997
   [junit4]   2> 2823733 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2823733 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2823734 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265
   [junit4]   2> 2823735 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/'
   [junit4]   2> 2823766 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/solr.xml
   [junit4]   2> 2823809 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2823810 T9918 oasc.CoreContainer.<init> New CoreContainer 2001606592
   [junit4]   2> 2823811 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/]
   [junit4]   2> 2823812 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2823812 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2823813 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2823813 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2823814 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2823814 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2823815 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2823815 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2823816 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2823838 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2823839 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2823839 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2823840 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2823840 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2823842 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2823854 T9984 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27f97b71 name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2823856 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2823861 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2823865 T9986 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c179983 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2823867 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2823880 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2824896 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60997_
   [junit4]   2> 2824903 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60997_
   [junit4]   2> 2824927 T9972 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2824929 T9940 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2824931 T9958 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2824935 T9945 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2824936 T9986 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2824954 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265
   [junit4]   2> 2825101 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/conf/
   [junit4]   2> 2825108 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2825109 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2825110 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2825424 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 2825426 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2825431 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:61000
   [junit4]   2> 2825431 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2825432 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2825433 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027
   [junit4]   2> 2825433 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/'
   [junit4]   2> 2825470 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/solr.xml
   [junit4]   2> 2825533 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2825535 T9918 oasc.CoreContainer.<init> New CoreContainer 1797760599
   [junit4]   2> 2825536 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/]
   [junit4]   2> 2825538 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2825538 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2825538 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2825539 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2825540 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2825540 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2825541 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2825541 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2825541 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2825571 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2825572 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2825573 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2825573 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2825575 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2825577 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825582 T9998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7819d0b6 name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825583 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825588 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2825592 T10000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a49ba5e name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2825593 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2825608 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2826622 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61000_
   [junit4]   2> 2826629 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61000_
   [junit4]   2> 2826656 T9986 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826658 T9972 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826659 T9940 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826661 T9958 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826663 T10000 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826666 T9945 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2826699 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027
   [junit4]   2> 2826815 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/conf/
   [junit4]   2> 2826819 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2826820 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2826821 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2827154 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 2827155 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2827160 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:61003
   [junit4]   2> 2827161 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2827161 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2827162 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738
   [junit4]   2> 2827162 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/'
   [junit4]   2> 2827192 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/solr.xml
   [junit4]   2> 2827243 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2827245 T9918 oasc.CoreContainer.<init> New CoreContainer 2104265141
   [junit4]   2> 2827245 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/]
   [junit4]   2> 2827246 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2827247 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2827247 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2827247 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2827248 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2827248 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2827249 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2827249 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2827249 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2827279 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2827280 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2827280 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2827280 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2827281 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2827283 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2827288 T10012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60c50a67 name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2827290 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2827296 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2827300 T10014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ab27c42 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2827301 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2827315 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2828331 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61003_
   [junit4]   2> 2828336 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61003_
   [junit4]   2> 2828365 T9972 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828369 T9958 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828371 T9940 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828373 T9986 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828375 T10014 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828377 T10000 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828378 T9945 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2828394 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738
   [junit4]   2> 2828537 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/conf/
   [junit4]   2> 2828545 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2828545 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2828546 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2828905 T9918 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 2828906 T9918 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2828911 T9918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:61006
   [junit4]   2> 2828911 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2828912 T9918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2828912 T9918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463
   [junit4]   2> 2828913 T9918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/'
   [junit4]   2> 2828942 T9918 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/solr.xml
   [junit4]   2> 2829012 T9918 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2829013 T9918 oasc.CoreContainer.<init> New CoreContainer 553880644
   [junit4]   2> 2829013 T9918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/]
   [junit4]   2> 2829014 T9918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2829015 T9918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2829015 T9918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2829016 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2829016 T9918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2829017 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2829017 T9918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2829018 T9918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2829018 T9918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2829039 T9918 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2829039 T9918 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2829040 T9918 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2829040 T9918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60984/solr
   [junit4]   2> 2829041 T9918 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2829042 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2829048 T10026 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fedd54a name:ZooKeeperConnection Watcher:127.0.0.1:60984 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2829050 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2829056 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2829060 T10028 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5da74cf6 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2829061 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2829074 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2830090 T9918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61006_
   [junit4]   2> 2830096 T9918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61006_
   [junit4]   2> 2830123 T9986 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830127 T10014 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830129 T9958 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830132 T9945 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830133 T9972 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830135 T10028 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830138 T10000 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830140 T9940 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2830170 T9918 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463
   [junit4]   2> 2830316 T9918 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/conf/
   [junit4]   2> 2830323 T9918 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2830324 T9918 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2830325 T9918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2830326 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2830332 T10031 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a180764 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2830334 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2830338 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2830348 T9918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2830352 T10033 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13419171 name:ZooKeeperConnection Watcher:127.0.0.1:60984/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2830353 T9918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2830356 T9918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2830369 T9965 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=2&name=testOverseerCol&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 2830372 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830376 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830381 T9942 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2830381 T9942 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2830384 T9942 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2830385 T9942 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 2830385 T9942 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 2830407 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830415 T9941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2830419 T9941 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 2830419 T9941 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2830433 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830438 T9972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830439 T10000 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830440 T9986 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830441 T10014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830442 T10033 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830442 T9945 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830442 T10031 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830445 T9940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830443 T10028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830453 T9958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830509 T9942 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 2830510 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:60987_
   [junit4]   2> 2830511 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:61000_
   [junit4]   2> 2830515 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:60991_
   [junit4]   2> 2830518 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:60997_
   [junit4]   2> 2830521 T9934 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 2830522 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:60994_
   [junit4]   2> 2830524 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:61006_
   [junit4]   2> 2830530 T9991 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 2830531 T9950 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 2830529 T9981 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 2830527 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830530 T9934 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830530 T9967 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 2830533 T9991 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830536 T9950 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830536 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:61003_
   [junit4]   2> 2830538 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830539 T9942 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:60987_
   [junit4]   2> 2830541 T9981 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830542 T10023 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 2830544 T9941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2830541 T9967 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830543 T10007 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 2830547 T9935 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 2830548 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60987",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60987_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830549 T10007 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830550 T10023 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830551 T9935 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2830561 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:61000",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:61000_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830569 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830573 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60991",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60991_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830581 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830585 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60997",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60997_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830594 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830598 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60994",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60994_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830608 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830614 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:61006",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:61006_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830624 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830630 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:61003",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:61003_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830644 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830651 T9941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60987",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60987_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2830663 T9940 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2830778 T9945 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830779 T9986 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830780 T10028 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830780 T10033 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830779 T9940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830779 T10014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830784 T10000 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830781 T9972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830785 T10031 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2830790 T9958 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2831535 T9934 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 2831536 T9991 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 2831541 T9950 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 2831542 T9991 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/testOverseerCol_shard1_replica2
   [junit4]   2> 2831543 T9991 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831545 T9950 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1
   [junit4]   2> 2831546 T9950 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831547 T9967 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 2831547 T9981 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 2831554 T9967 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/testOverseerCol_shard3_replica1
   [junit4]   2> 2831555 T9967 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831556 T9981 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/testOverseerCol_shard2_replica2
   [junit4]   2> 2831557 T9981 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831557 T10007 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 2831547 T9934 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/testOverseerCol_shard1_replica1
   [junit4]   2> 2831558 T10023 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 2831560 T10007 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/testOverseerCol_shard4_replica1
   [junit4]   2> 2831560 T10007 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831561 T10023 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/testOverseerCol_shard3_replica2
   [junit4]   2> 2831561 T10023 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831562 T9934 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831553 T9935 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 2831565 T10007 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831565 T10007 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831566 T9935 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/testOverseerCol_shard4_replica2
   [junit4]   2> 2831567 T9935 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2831567 T10023 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831568 T10023 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831564 T9991 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831563 T9981 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831571 T9981 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831572 T9935 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831572 T9935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831573 T9991 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831575 T10023 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831576 T10023 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391831093463/testOverseerCol_shard3_replica2/'
   [junit4]   2> 2831562 T9967 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831578 T9967 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831581 T9967 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831581 T9967 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391831086565/testOverseerCol_shard3_replica1/'
   [junit4]   2> 2831562 T9950 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831569 T10007 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831585 T10007 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391831091738/testOverseerCol_shard4_replica1/'
   [junit4]   2> 2831586 T9934 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2831586 T9981 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831588 T9981 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391831088265/testOverseerCol_shard2_replica2/'
   [junit4]   2> 2831589 T9934 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831594 T9950 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2831599 T9935 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831599 T9935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/testOverseerCol_shard4_replica2/'
   [junit4]   2> 2831603 T9991 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831603 T9991 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1391831090027/testOverseerCol_shard1_replica2/'
   [junit4]   2> 2831609 T9950 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831610 T9950 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/'
   [junit4]   2> 2831610 T9934 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2831611 T9934 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391831084117/testOverseerCol_shard1_replica1/'
   [junit4]   2> 2831746 T9967 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831825 T9935 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831834 T9991 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831849 T10023 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831870 T10007 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831876 T9981 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831887 T9950 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831888 T9934 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 2831996 T9967 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832003 T9967 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832037 T9967 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 2832048 T9935 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832052 T10007 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832052 T9991 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832054 T9934 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832060 T9935 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832065 T9991 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832073 T9935 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 2832084 T9981 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832085 T10007 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832085 T9934 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832088 T9981 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832091 T9991 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 2832104 T9981 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 2832109 T9950 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832136 T10023 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2832140 T9934 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 2832152 T10007 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 2832153 T9950 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832166 T10023 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2832199 T9950 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 2832208 T10023 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 2834026 T9935 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834053 T9935 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834054 T9935 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834057 T9950 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834075 T9950 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834077 T9950 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834097 T9934 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834099 T9934 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834103 T9934 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834104 T9950 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834110 T9950 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834122 T9935 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834130 T9934 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834142 T9935 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834145 T9934 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834247 T9967 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834249 T9967 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834250 T9967 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834269 T9967 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834276 T9967 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834289 T10007 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834290 T10007 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834292 T10007 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834308 T10007 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834313 T10007 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834341 T9991 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834343 T9991 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834347 T9991 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834382 T9991 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834409 T9981 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834422 T9981 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834425 T9981 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834427 T9991 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834443 T9981 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834444 T10023 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834445 T10023 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2834446 T10023 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2834447 T9981 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834462 T10023 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2834470 T10023 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2834471 T9950 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2834487 T9950 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2834513 T9950 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2834544 T9950 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2834545 T9950 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2834546 T9950 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2834559 T9950 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2834561 T9950 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2834561 T9950 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2834563 T9950 oasc.SolrCore.<init> [testOverseerCol_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/data/
   [junit4]   2> 2834563 T9950 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b7689bc
   [junit4]   2> 2834566 T9950 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/data
   [junit4]   2> 2834566 T9950 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/data/index/
   [junit4]   2> 2834567 T9950 oasc.SolrCore.initIndex WARN [testOverseerCol_shard2_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2834569 T9950 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391831084879/testOverseerCol_shard2_replica1/data/index
   [junit4]   2> 2834570 T9950 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=64415187, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2834573 T9950

[...truncated too long message...]

for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2844216 T10028 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=ja, timezone=America/Indiana/Tell_City
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=1,free=35255232,total=212180992
   [junit4]   2> NOTE: All tests run in this JVM: [TestRecoveryHdfs, TestShardHandlerFactory, AlternateDirectoryTest, CoreContainerCoreInitFailuresTest, QueryEqualityTest, DateFieldTest, TestValueSourceCache, XmlUpdateRequestHandlerTest, DocValuesTest, TestDynamicFieldCollectionResource, BlockDirectoryTest, TestFunctionQuery, SliceStateUpdateTest, TestFieldTypeResource, DirectSolrSpellCheckerTest, HdfsBasicDistributedZkTest, MBeansHandlerTest, TestSchemaSimilarityResource, InfoHandlerTest, TestLFUCache, TestOverriddenPrefixQueryForCustomFieldType, TestSolrJ, DistanceFunctionTest, SystemInfoHandlerTest, TestDistributedMissingSort, TermsComponentTest, TestCloudManagedSchema, TestLazyCores, PathHierarchyTokenizerFactoryTest, FileBasedSpellCheckerTest, TestPartialUpdateDeduplication, AnalysisErrorHandlingTest, TestSolrQueryParserDefaultOperatorResource, TestFaceting, TestSolrDeletionPolicy2, HdfsCollectionsAPIDistributedZkTest, TestFieldResource, TestNonNRTOpen, QueryResultKeyTest, CoreAdminCreateDiscoverTest, TestHighFrequencyDictionaryFactory, TriLevelCompositeIdRoutingTest, TestSolr4Spatial, TestNRTOpen, ClusterStateUpdateTest, SuggesterTSTTest, DeleteReplicaTest, TestJoin, TestHashPartitioner, CursorMarkTest, HdfsRecoveryZkTest, ShardRoutingTest, DistribCursorPagingTest, SliceStateTest, TestUniqueKeyFieldResource, BadComponentTest, TestInitQParser, DistributedSuggestComponentTest, TestReload, MinimalSchemaTest, TestCursorMarkWithoutUniqueKey, UnloadDistributedZkTest, SolrInfoMBeanTest, SuggesterWFSTTest, TestUpdate, MoreLikeThisHandlerTest, ConvertedLegacyTest, TestPHPSerializedResponseWriter, CopyFieldTest, PrimitiveFieldTypeTest, ZkNodePropsTest, SOLR749Test, DebugComponentTest, TestRemoteStreaming, DirectUpdateHandlerOptimizeTest, BasicFunctionalityTest, TestReloadAndDeleteDocs, TestFileDictionaryLookup, TestStressLucene, SuggesterFSTTest, SyncSliceTest, TestNoOpRegenerator, TestUtils, TestComponentsName, ReturnFieldsTest, TestPerFieldSimilarity, TestClassNameShortening, NotRequiredUniqueKeyTest, TestLMJelinekMercerSimilarityFactory, TestMultiCoreConfBootstrap, CoreAdminHandlerTest, URLClassifyProcessorTest, CurrencyFieldXmlFileTest, RequiredFieldsTest, TestWriterPerf, TestBadConfig, TestCustomSort, DirectSolrConnectionTest, AliasIntegrationTest, TestBM25SimilarityFactory, TestStressVersions, DOMUtilTest, HdfsLockFactoryTest, HdfsSyncSliceTest, TestImplicitCoreProperties, PrimUtilsTest, SolrXmlInZkTest, StandardRequestHandlerTest, TestDocumentBuilder, SolrIndexSplitterTest, DocumentBuilderTest, TestReversedWildcardFilterFactory, BasicZkTest, StressHdfsTest, TestStandardQParsers, DistributedDebugComponentTest, DocValuesMultiTest, TestRangeQuery, AutoCommitTest, TestSolrXmlPersistor, SpellCheckComponentTest, LegacyHTMLStripCharFilterTest, TestIndexSearcher, OpenExchangeRatesOrgProviderTest, ShardRoutingCustomTest, DefaultValueUpdateProcessorTest, TestAtomicUpdateErrorCases, TestStressReorder, TestDefaultSimilarityFactory, ParsingFieldUpdateProcessorsTest, BJQParserTest, ScriptEngineTest, TestSearchPerf, TestInfoStreamLogging, TestTrie, TestSimpleQParserPlugin, SolrRequestParserTest, TestQueryTypes, FastVectorHighlighterTest, TestDocBasedVersionConstraints, MigrateRouteKeyTest, SortByFunctionTest, SignatureUpdateProcessorFactoryTest, UpdateParamsTest, TestRecovery, UUIDFieldTest, CSVRequestHandlerTest, SpellPossibilityIteratorTest, SpellingQueryConverterTest, TermVectorComponentTest, TestCollapseQParserPlugin, BasicDistributedZk2Test, ChaosMonkeySafeLeaderTest, TestDistributedGrouping, TestSweetSpotSimilarityFactory, DistributedSpellCheckComponentTest, FullSolrCloudDistribCmdsTest, WordBreakSolrSpellCheckerTest, HighlighterTest, TestExtendedDismaxParser, TestJmxMonitoredMap, TestRandomMergePolicy, TestFieldCollectionResource, CoreMergeIndexesAdminHandlerTest, TestQueryUtils, UpdateRequestProcessorFactoryTest, FieldMutatingUpdateProcessorTest, BadCopyFieldTest, FileUtilsTest, TestSystemIdResolver, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, RecoveryZkTest, TestReplicationHandler, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, ZkCLITest, TestDistributedSearch, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, HardAutoCommitTest, DistributedTermsComponentTest, TestCoreContainer, SimpleFacetsTest, SolrCoreTest, StatsComponentTest, TestGroupingSearch, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, TestSort, TestFiltering, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, SoftAutoCommitTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, SuggesterTest, SpatialFilterTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, RequestHandlersTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, JSONWriterTest, TestCSVResponseWriter, QueryParsingTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, SearchHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, TestQuerySenderListener, TestSolrIndexConfig, TestQuerySenderNoQuery, ResponseLogComponentTest, SolrIndexConfigTest, TestStressRecovery, TestMergePolicyConfig, MultiTermTest, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestDFRSimilarityFactory, TestNumberUtils, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, PluginInfoTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestSolrXMLSerializer, PreAnalyzedFieldTest, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, ClusterStateTest, TestLRUCache, CircularListTest, TestRTGBase, CursorPagingTest, SolrTestCaseJ4Test, TestCollationKeyRangeQueries, AssignTest, ConnectionManagerTest, CustomCollectionTest, DeleteShardTest, OverseerRolesTest]
   [junit4] Completed in 26.38s, 1 test, 1 error <<< FAILURES!

[...truncated 121 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 358 suites, 1554 tests, 1 error, 40 ignored (6 assumptions)

Total time: 89 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure