You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/03 05:29:23 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1289 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1289/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=1228, name=Overseer-91186736294920206-127.0.0.1:49523_-n_0000000005, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1228, name=Overseer-91186736294920206-127.0.0.1:49523_-n_0000000005, state=RUNNABLE, group=Overseer collection creation process.]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([CF65042CE516355A]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:202)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:192)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:193)
	... 1 more




Build Log:
[...truncated 9915 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 439352 T1072 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 439359 T1072 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391399174410
   [junit4]   2> 439360 T1072 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 439361 T1073 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 439365 T1072 oasc.ZkTestServer.run start zk server on port:49504
   [junit4]   2> 439384 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 439396 T1079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7079e04c name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 439396 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 439397 T1072 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 439410 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 439412 T1081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a9fea98 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 439413 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 439413 T1072 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 439423 T1072 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 439431 T1072 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 439440 T1072 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 439448 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 439449 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 439461 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 439462 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 439473 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 439473 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 439483 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 439484 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 439494 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 439495 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 439504 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 439504 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 439523 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 439524 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 439535 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 439536 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 439551 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 439552 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 439562 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 439563 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 439571 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 439571 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 439984 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 439993 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49507
   [junit4]   2> 439993 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/'
   [junit4]   2> 440021 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/solr.xml
   [junit4]   2> 440098 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 440107 T1072 oasc.CoreContainer.<init> New CoreContainer 527817310
   [junit4]   2> 440107 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/]
   [junit4]   2> 440108 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 440108 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 440121 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 440121 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 440121 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 440122 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 440122 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 440123 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440129 T1092 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72536174 name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440131 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440140 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440145 T1094 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32de1a32 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440146 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440150 T1072 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 440161 T1072 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 440170 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 440175 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49507_
   [junit4]   2> 440178 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49507_
   [junit4]   2> 440188 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 440196 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 440213 T1072 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49507_
   [junit4]   2> 440213 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 440224 T1072 oasc.Overseer.start Overseer (id=91186736294920195-127.0.0.1:49507_-n_0000000000) starting
   [junit4]   2> 440238 T1072 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 440260 T1096 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 440265 T1072 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 440265 T1096 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 440274 T1072 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 440280 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 440292 T1095 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 440305 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632
   [junit4]   2> 440411 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/conf/
   [junit4]   2> 440416 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 440418 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 440418 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 440420 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440425 T1099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d655ccd name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440426 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440431 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 440439 T1072 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 440798 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 440799 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 440803 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49511
   [junit4]   2> 440804 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/'
   [junit4]   2> 440837 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/solr.xml
   [junit4]   2> 440918 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 440919 T1072 oasc.CoreContainer.<init> New CoreContainer 279886935
   [junit4]   2> 440919 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/]
   [junit4]   2> 440921 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 440921 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 440935 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 440936 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 440936 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 440937 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 440937 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 440939 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440943 T1110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fcefc4f name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440944 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440950 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440955 T1112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6844f2eb name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440955 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440968 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 441978 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49511_
   [junit4]   2> 441981 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49511_
   [junit4]   2> 441992 T1112 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 441992 T1099 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 441996 T1094 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 442006 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490
   [junit4]   2> 442120 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/conf/
   [junit4]   2> 442127 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 442127 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 442127 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 442471 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 442472 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 442476 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49514
   [junit4]   2> 442478 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 442478 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 442479 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179
   [junit4]   2> 442479 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/'
   [junit4]   2> 442509 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/solr.xml
   [junit4]   2> 442588 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 442588 T1072 oasc.CoreContainer.<init> New CoreContainer 141531096
   [junit4]   2> 442589 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/]
   [junit4]   2> 442590 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 442590 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 442592 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 442592 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 442593 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 442603 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 442604 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 442604 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 442604 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 442604 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 442606 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 442610 T1124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5659ea3b name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442611 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 442621 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 442625 T1126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35a5c518 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442626 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 442640 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 443652 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49514_
   [junit4]   2> 443655 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49514_
   [junit4]   2> 443664 T1126 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443665 T1112 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443665 T1094 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443666 T1099 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443683 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179
   [junit4]   2> 443795 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/conf/
   [junit4]   2> 443802 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 443803 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 443803 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 444156 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 444158 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 444162 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49517
   [junit4]   2> 444164 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 444164 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 444164 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856
   [junit4]   2> 444165 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/'
   [junit4]   2> 444198 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/solr.xml
   [junit4]   2> 444276 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 444277 T1072 oasc.CoreContainer.<init> New CoreContainer 1017154163
   [junit4]   2> 444277 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/]
   [junit4]   2> 444278 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 444278 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 444292 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 444293 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 444293 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 444294 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 444294 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 444295 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444299 T1138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fe1a31d name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444300 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444307 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444311 T1140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f28bac name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444312 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444326 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 445340 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49517_
   [junit4]   2> 445345 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49517_
   [junit4]   2> 445366 T1112 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445368 T1094 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445370 T1099 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445370 T1126 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445372 T1140 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445408 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856
   [junit4]   2> 445515 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/conf/
   [junit4]   2> 445522 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 445523 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 445524 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 445958 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 445960 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 445969 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49520
   [junit4]   2> 445972 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 445973 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 445990 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576
   [junit4]   2> 445990 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/'
   [junit4]   2> 446023 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/solr.xml
   [junit4]   2> 446092 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 446093 T1072 oasc.CoreContainer.<init> New CoreContainer 509384364
   [junit4]   2> 446093 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/]
   [junit4]   2> 446095 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 446096 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 446096 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 446097 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 446097 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 446110 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 446111 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 446112 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 446112 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 446113 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 446114 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446118 T1152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50a805c6 name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446120 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446127 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446132 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b238cce name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446133 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446146 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 447162 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49520_
   [junit4]   2> 447166 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49520_
   [junit4]   2> 447181 T1099 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447182 T1094 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1126 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1140 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1112 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447184 T1154 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447198 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576
   [junit4]   2> 447365 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/conf/
   [junit4]   2> 447375 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 447376 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 447376 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 447749 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 447749 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 447756 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49523
   [junit4]   2> 447758 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 447758 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 447758 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428
   [junit4]   2> 447759 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/'
   [junit4]   2> 447788 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/solr.xml
   [junit4]   2> 447860 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 447861 T1072 oasc.CoreContainer.<init> New CoreContainer 1780100759
   [junit4]   2> 447861 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/]
   [junit4]   2> 447862 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 447865 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 447877 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 447878 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 447878 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 447878 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 447879 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 447880 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447886 T1166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ce8f1ec name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447887 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447895 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447898 T1168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be0127b name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447898 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447912 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448923 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49523_
   [junit4]   2> 448926 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49523_
   [junit4]   2> 448937 T1099 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1094 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1154 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1168 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1112 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1126 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1140 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448957 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428
   [junit4]   2> 449117 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/conf/
   [junit4]   2> 449125 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 449126 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 449127 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 449484 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 449485 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 449489 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49526
   [junit4]   2> 449491 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 449492 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 449492 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178
   [junit4]   2> 449493 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/'
   [junit4]   2> 449525 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/solr.xml
   [junit4]   2> 449600 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 449600 T1072 oasc.CoreContainer.<init> New CoreContainer 1860417090
   [junit4]   2> 449601 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/]
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 449604 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 449604 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 449620 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 449620 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 449621 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 449621 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 449621 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 449623 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449628 T1180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27d2dfee name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449629 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449636 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449641 T1182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35e406c1 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449642 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449658 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 450674 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49526_
   [junit4]   2> 450679 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49526_
   [junit4]   2> 450695 T1099 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450696 T1140 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1154 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1126 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1182 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1094 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1112 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1168 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450715 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178
   [junit4]   2> 450877 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/conf/
   [junit4]   2> 450884 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 450885 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 450885 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450887 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451594 T1185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2041a823 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451596 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451605 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451620 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451627 T1187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20ea3120 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451628 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451632 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451647 T1089 oasha.CollectionsHandler.handleCreateAction Creating Collection : wt=javabin&replicationFactor=2&maxShardsPerNode=2&name=testOverseerCol&version=2&numShards=4&action=CREATE
   [junit4]   2> 451650 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451654 T1096 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 451654 T1096 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 451656 T1096 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 451656 T1096 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 451657 T1096 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 451668 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451673 T1095 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451675 T1095 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 451675 T1095 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 451681 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451685 T1168 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1126 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1182 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451771 T1096 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 451771 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:49514_
   [junit4]   2> 451772 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:49526_
   [junit4]   2> 451774 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:49520_
   [junit4]   2> 451775 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:49517_
   [junit4]   2> 451778 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:49523_
   [junit4]   2> 451779 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:49511_
   [junit4]   2> 451782 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:49507_
   [junit4]   2> 451786 T1122 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 451787 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:49514_
   [junit4]   2> 451788 T1136 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 451793 T1147 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 451796 T1178 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 451797 T1157 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 451797 T1117 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 451799 T1122 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451803 T1090 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 451801 T1106 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 451804 T1136 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451801 T1147 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451801 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451805 T1117 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451805 T1178 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451807 T1106 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451803 T1157 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451807 T1090 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451811 T1095 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451813 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49514",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49514_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451820 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451823 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49520",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49520_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451831 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451834 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49523",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49523_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451845 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451849 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49517",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49517_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451856 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451859 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49526",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49526_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451867 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451871 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49514",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49514_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451879 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451882 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49511",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49511_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451890 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451898 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49507",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49507_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451906 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452015 T1112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452015 T1168 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452015 T1094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1182 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1126 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452804 T1122 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 452806 T1136 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 452808 T1147 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 452810 T1136 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/testOverseerCol_shard2_replica2
   [junit4]   2> 452810 T1136 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452811 T1117 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 452812 T1122 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1
   [junit4]   2> 452815 T1122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452817 T1117 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard4_replica2
   [junit4]   2> 452819 T1117 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452812 T1106 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 452823 T1122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452823 T1122 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452819 T1136 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452825 T1136 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452815 T1090 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 452827 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/testOverseerCol_shard2_replica2/'
   [junit4]   2> 452828 T1090 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/testOverseerCol_shard4_replica1
   [junit4]   2> 452829 T1090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452830 T1122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/'
   [junit4]   2> 452814 T1157 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 452813 T1178 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 452833 T1157 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1
   [junit4]   2> 452835 T1157 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452835 T1178 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2
   [junit4]   2> 452843 T1178 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452813 T1147 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/testOverseerCol_shard2_replica1
   [junit4]   2> 452844 T1147 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452824 T1117 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452848 T1117 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452849 T1090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452849 T1090 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452850 T1147 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452850 T1147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452853 T1117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard4_replica2/'
   [junit4]   2> 452824 T1106 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2
   [junit4]   2> 452857 T1106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452857 T1178 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452858 T1178 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452858 T1157 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452859 T1157 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452861 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/testOverseerCol_shard2_replica1/'
   [junit4]   2> 452878 T1106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452879 T1106 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452892 T1090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/testOverseerCol_shard4_replica1/'
   [junit4]   2> 452906 T1157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/'
   [junit4]   2> 452917 T1178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2/'
   [junit4]   2> 452938 T1106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/'
   [junit4]   2> 453083 T1117 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453089 T1178 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453103 T1106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453135 T1147 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453187 T1157 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453191 T1136 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453210 T1090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453322 T1122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453368 T1117 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453372 T1117 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453380 T1106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453390 T1117 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 453393 T1147 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453399 T1178 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453400 T1106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453412 T1106 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 453418 T1178 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453437 T1178 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 453442 T1147 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453453 T1147 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 453502 T1136 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453512 T1136 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453521 T1136 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 453582 T1090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453585 T1090 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453601 T1122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453606 T1122 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453615 T1122 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 453625 T1090 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 453683 T1157 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453688 T1157 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453700 T1157 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 455726 T1122 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455758 T1122 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 455784 T1122 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 455862 T1122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455870 T1122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455890 T1122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455908 T1106 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455914 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455918 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455919 T1122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455943 T1178 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455950 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455951 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455951 T1122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 455951 T1122 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/
   [junit4]   2> 455952 T1122 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fe1526c
   [junit4]   2> 455953 T1122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data
   [junit4]   2> 455954 T1122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index/
   [junit4]   2> 455954 T1122 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 455955 T1122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index
   [junit4]   2> 455955 T1122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=82.4609375, floorSegmentMB=1.365234375, forceMergeDeletesPctAllowed=7.33938087738324, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20062408356616307
   [junit4]   2> 455956 T1122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@586fb801 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31dd55ff),segFN=segments_1,generation=1}
   [junit4]   2> 455957 T1122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455970 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 455971 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 455971 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 455973 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455973 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455974 T1122 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 455974 T1122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 455988 T1122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 455989 T1122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 455990 T1122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 455991 T1122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 455992 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 455992 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 455993 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 456014 T1106 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456021 T1106 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456029 T1178 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456033 T1178 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456056 T1122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 456062 T1122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 456063 T1122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 456064 T1122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=88.7353515625, floorSegmentMB=1.6591796875, forceMergeDeletesPctAllowed=16.59613178711819, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.28160697878444846
   [junit4]   2> 456089 T1106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456096 T1178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456097 T1122 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@586fb801 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31dd55ff),segFN=segments_1,generation=1}
   [junit4]   2> 456097 T1122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 456098 T1122 oass.SolrIndexSearcher.<init> Opening Searcher@377b6d55 main
   [junit4]   2> 456106 T1196 oasc.SolrCore.registerSearcher [testOverseerCol_shard1_replica1] Registered new searcher Searcher@377b6d55 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 456112 T1106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456116 T1178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456124 T1106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456127 T1117 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 456133 T1178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456140 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456147 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456147 T1106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456150 T1117 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456154 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456155 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456155 T1178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456157 T1122 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard1_replica1
   [junit4]   2> 456158 T1122 oasc.ZkController.register Register replica - core:testOverseerCol_shard1_replica1 address:http://127.0.0.1:49514 collection:testOverseerCol shard:shard1
   [junit4]   2> 456160 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456160 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456160 T1106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 456161 T1106 oasc.SolrCore.<init> [testOverseerCol_shard3_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/data/
   [junit4]   2> 456161 T1106 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fe1526c
   [junit4]   2> 456162 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456163 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456163 T1106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/data
   [junit4]   2> 456165 T1117 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456163 T1178 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 456170 T1178 oasc.SolrCore.<init> [testOverseerCol_shard1_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2/, dataDir=../../../../../../../../../Users/jenkins/works

[...truncated too long message...]

464653 T1168 C448 P49523 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 464653 T1168 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 464653 T1168 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49523/testOverseerCol_shard3_replica1/ shard3
   [junit4]   2> 464654 T1168 oasc.SolrCore.close [testOverseerCol_shard3_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@34ff584e
   [junit4]   2> 464669 T1168 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 464669 T1168 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 464669 T1168 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 464670 T1168 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 464671 T1168 oasc.SolrCore.closeSearcher [testOverseerCol_shard3_replica1] Closing main searcher on request.
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data;done=false>>]
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index;done=false>>]
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index
   [junit4]   2> 464672 T1168 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leaders/shard3
   [junit4]   2> 464673 T1168 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 464673 T1168 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard3/election/91186736294920206-core_node3-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 464673 T1168 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=es_EC, timezone=Australia/Queensland
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=5,free=33196904,total=277872640
   [junit4]   2> NOTE: All tests run in this JVM: [BlockCacheTest, OverseerCollectionProcessorTest, MBeansHandlerTest, SignatureUpdateProcessorFactoryTest, TestDocSet, PreAnalyzedFieldTest, HdfsRecoveryZkTest, ResponseLogComponentTest, FullSolrCloudDistribCmdsTest, TermVectorComponentTest, TestExtendedDismaxParser, TestStressReorder, PolyFieldTest, InfoHandlerTest, SolrInfoMBeanTest, TestCopyFieldCollectionResource, RecoveryZkTest, TestLRUCache, TestFastWriter, MinimalSchemaTest, TestWriterPerf, TestDocBasedVersionConstraints, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, HdfsDirectoryTest, TestUniqueKeyFieldResource, DistanceFunctionTest, CurrencyFieldOpenExchangeTest, EnumFieldTest, TestElisionMultitermQuery, TestSolrQueryParserResource, DirectSolrSpellCheckerTest, DateMathParserTest, NoCacheHeaderTest, RequestHandlersTest, DistribCursorPagingTest, TestPerFieldSimilarity, TestNRTOpen, DefaultValueUpdateProcessorTest, LeaderElectionIntegrationTest, TestLMJelinekMercerSimilarityFactory, XmlUpdateRequestHandlerTest, CurrencyFieldXmlFileTest, ConnectionManagerTest, OverseerRolesTest]
   [junit4] Completed in 25.48s, 1 test, 1 error <<< FAILURES!

[...truncated 984 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 error, 40 ignored (6 assumptions)

Total time: 79 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure