You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/31 11:45:30 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0) - Build # 9953 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9953/
Java: 32bit/jdk1.8.0 -server -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=4393, name=Overseer-91505217006010374-127.0.0.1:44654_-n_0000000011, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4393, name=Overseer-91505217006010374-127.0.0.1:44654_-n_0000000011, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([725FF5CB1B30450C:F3B97BD36C6F2530]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([725FF5CB1B30450C]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:275)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000006
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:276)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:273)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:206)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:258)
	... 1 more




Build Log:
[...truncated 11553 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408
   [junit4]   2> 1110938 T4214 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1110941 T4214 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1110941 T4214 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1110942 T4215 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1111042 T4214 oasc.ZkTestServer.run start zk server on port:36232
   [junit4]   2> 1111043 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112311 T4218 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1267ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 1112312 T4221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@abd906 name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112312 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112313 T4214 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1112316 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112318 T4223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c96c2 name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112318 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112319 T4214 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1112324 T4214 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1112325 T4214 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1112327 T4214 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1112328 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1112329 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1112332 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1112332 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1112334 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1112335 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1112337 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1112337 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1112339 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1112339 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1112342 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1112342 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1112345 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1112345 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1112347 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1112347 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1112349 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1112349 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1112350 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1112351 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1112352 T4214 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1112353 T4214 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1112593 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1112612 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51748
   [junit4]   2> 1112612 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1112613 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1112613 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825
   [junit4]   2> 1112614 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825/'
   [junit4]   2> 1112625 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825/solr.xml
   [junit4]   2> 1112632 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825
   [junit4]   2> 1112633 T4214 oasc.CoreContainer.<init> New CoreContainer 24388798
   [junit4]   2> 1112633 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825/]
   [junit4]   2> 1112634 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1112634 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1112634 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1112635 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1112635 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1112635 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1112635 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1112636 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1112636 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1112637 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1112637 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1112637 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1112638 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1112638 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1112639 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112641 T4234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@163b331 name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112641 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112643 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112643 T4236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c23ac7 name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112643 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112645 T4214 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1112646 T4214 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1112648 T4214 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1112649 T4214 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1112651 T4214 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1112652 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1112653 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51748_
   [junit4]   2> 1112654 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51748_
   [junit4]   2> 1112655 T4214 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1112659 T4214 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1112662 T4214 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51748_
   [junit4]   2> 1112662 T4214 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1112663 T4214 oasc.Overseer.start Overseer (id=91505217006010371-127.0.0.1:51748_-n_0000000000) starting
   [junit4]   2> 1112665 T4214 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1112669 T4238 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1112669 T4214 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1112669 T4238 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1112672 T4214 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1112674 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1112676 T4237 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1112677 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825
   [junit4]   2> 1112683 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825/conf/
   [junit4]   2> 1112812 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1112813 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1112814 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1112815 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112816 T4241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ea3af3 name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112816 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112817 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1112818 T4214 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1113205 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1113206 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1113208 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44654
   [junit4]   2> 1113209 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1113209 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1113210 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288
   [junit4]   2> 1113210 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288/'
   [junit4]   2> 1113228 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288/solr.xml
   [junit4]   2> 1113267 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288
   [junit4]   2> 1113268 T4214 oasc.CoreContainer.<init> New CoreContainer 23775523
   [junit4]   2> 1113268 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288/]
   [junit4]   2> 1113269 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1113270 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1113271 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1113271 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1113272 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1113273 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1113273 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1113274 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1113275 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1113276 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1113277 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1113277 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1113277 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1113278 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1113279 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1113280 T4252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@191f3b7 name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1113280 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1113282 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1113283 T4254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11089b8 name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1113283 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1113286 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1114288 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44654_
   [junit4]   2> 1114290 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44654_
   [junit4]   2> 1114291 T4236 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1114291 T4241 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1114292 T4254 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1114294 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288
   [junit4]   2> 1114303 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288/conf/
   [junit4]   2> 1114431 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1114432 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1114432 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1114677 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1114678 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1114680 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58991
   [junit4]   2> 1114680 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1114681 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1114682 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903
   [junit4]   2> 1114682 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903/'
   [junit4]   2> 1114698 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903/solr.xml
   [junit4]   2> 1114706 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903
   [junit4]   2> 1114706 T4214 oasc.CoreContainer.<init> New CoreContainer 22841167
   [junit4]   2> 1114707 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903/]
   [junit4]   2> 1114708 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1114708 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1114708 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1114709 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1114709 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1114710 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1114710 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1114710 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1114711 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1114711 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1114711 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1114712 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1114712 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1114713 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1114713 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1114715 T4266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bf95b name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1114715 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1114717 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1114718 T4268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c584da name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1114718 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1114723 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1115727 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58991_
   [junit4]   2> 1115728 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58991_
   [junit4]   2> 1115729 T4241 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1115729 T4254 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1115729 T4268 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1115729 T4236 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1115733 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903
   [junit4]   2> 1115740 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903/conf/
   [junit4]   2> 1115866 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1115867 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1115867 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1116105 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1116106 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1116109 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52924
   [junit4]   2> 1116110 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1116110 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1116111 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338
   [junit4]   2> 1116111 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338/'
   [junit4]   2> 1116122 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338/solr.xml
   [junit4]   2> 1116129 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338
   [junit4]   2> 1116129 T4214 oasc.CoreContainer.<init> New CoreContainer 23371672
   [junit4]   2> 1116130 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338/]
   [junit4]   2> 1116130 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1116130 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1116131 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1116131 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1116131 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1116132 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1116132 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1116132 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1116133 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1116133 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1116133 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1116134 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1116134 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1116134 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1116135 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1116136 T4280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0a09b name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1116136 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1116138 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1116139 T4282 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d285c name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1116139 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1116143 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1117144 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52924_
   [junit4]   2> 1117145 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52924_
   [junit4]   2> 1117147 T4241 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1117147 T4254 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1117147 T4236 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1117147 T4268 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1117148 T4282 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1117150 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338
   [junit4]   2> 1117158 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338/conf/
   [junit4]   2> 1117346 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1117347 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1117348 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1117636 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1117636 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1117639 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54702
   [junit4]   2> 1117640 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1117640 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1117641 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819
   [junit4]   2> 1117641 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819/'
   [junit4]   2> 1117652 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819/solr.xml
   [junit4]   2> 1117674 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819
   [junit4]   2> 1117675 T4214 oasc.CoreContainer.<init> New CoreContainer 33103474
   [junit4]   2> 1117676 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819/]
   [junit4]   2> 1117676 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1117676 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1117677 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1117677 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1117677 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1117678 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1117678 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1117678 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1117678 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1117679 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1117679 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1117680 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1117680 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1117680 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1117681 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1117683 T4294 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f9bc72 name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1117683 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1117685 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1117686 T4296 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c755f name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1117686 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1117690 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1118691 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54702_
   [junit4]   2> 1118693 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54702_
   [junit4]   2> 1118694 T4254 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118695 T4241 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118695 T4296 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118695 T4236 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118695 T4282 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118695 T4268 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1118697 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819
   [junit4]   2> 1118702 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819/conf/
   [junit4]   2> 1118808 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1118809 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1118809 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1119072 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 1119072 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1119074 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53551
   [junit4]   2> 1119075 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1119075 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1119076 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280
   [junit4]   2> 1119076 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/'
   [junit4]   2> 1119087 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/solr.xml
   [junit4]   2> 1119094 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280
   [junit4]   2> 1119094 T4214 oasc.CoreContainer.<init> New CoreContainer 12253565
   [junit4]   2> 1119095 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/]
   [junit4]   2> 1119095 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1119096 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1119096 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1119096 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1119096 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1119097 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1119097 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1119097 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1119098 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1119098 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1119099 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1119099 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1119099 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1119100 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1119100 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1119101 T4308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c35ca name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1119101 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1119106 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1119107 T4310 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c505c name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1119107 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1119111 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1120113 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53551_
   [junit4]   2> 1120115 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53551_
   [junit4]   2> 1120117 T4310 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4296 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4254 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4236 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4268 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4241 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120117 T4282 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1120119 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280
   [junit4]   2> 1120124 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/conf/
   [junit4]   2> 1120220 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1120221 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1120221 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1120459 T4214 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 1120460 T4214 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1120462 T4214 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59669
   [junit4]   2> 1120463 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1120463 T4214 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1120463 T4214 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692
   [junit4]   2> 1120464 T4214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692/'
   [junit4]   2> 1120475 T4214 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692/solr.xml
   [junit4]   2> 1120481 T4214 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692
   [junit4]   2> 1120482 T4214 oasc.CoreContainer.<init> New CoreContainer 27420011
   [junit4]   2> 1120482 T4214 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692/]
   [junit4]   2> 1120483 T4214 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1120483 T4214 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1120483 T4214 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1120483 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1120484 T4214 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1120484 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1120484 T4214 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1120485 T4214 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1120485 T4214 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1120485 T4214 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1120486 T4214 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1120486 T4214 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1120487 T4214 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36232/solr
   [junit4]   2> 1120487 T4214 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1120488 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1120489 T4322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a522e5 name:ZooKeeperConnection Watcher:127.0.0.1:36232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1120489 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1120491 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1120491 T4324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f7d4df name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1120492 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1120495 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1121497 T4214 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59669_
   [junit4]   2> 1121498 T4214 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59669_
   [junit4]   2> 1121500 T4241 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4268 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4236 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4324 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4296 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4310 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4254 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121500 T4282 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1121503 T4214 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692
   [junit4]   2> 1121509 T4214 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692/conf/
   [junit4]   2> 1121715 T4214 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1121723 T4214 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1121723 T4214 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1121725 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1121725 T4327 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eb1317 name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1121725 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1121726 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1121728 T4214 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1121729 T4329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ea19c name:ZooKeeperConnection Watcher:127.0.0.1:36232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1121729 T4214 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1121730 T4214 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1121733 T4275 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=testOverseerCol&action=CREATE&numShards=4&wt=javabin&version=2
   [junit4]   2> 1121735 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121735 T4238 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1121735 T4238 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1121736 T4238 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1121736 T4238 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 1121737 T4238 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 1121738 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121739 T4237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1121740 T4237 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 1121740 T4237 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 1121741 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121742 T4241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4296 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4310 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121742 T4327 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121744 T4282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121744 T4324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121744 T4268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121839 T4238 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2, shard3, shard4] , replicationFactor : 2
   [junit4]   2> 1121839 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:53551_
   [junit4]   2> 1121840 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:58991_
   [junit4]   2> 1121842 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:44654_
   [junit4]   2> 1121842 T4305 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard1_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1121843 T4264 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard1_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1121843 T4305 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1121843 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:54702_
   [junit4]   2> 1121844 T4248 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard2_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1121845 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121845 T4305 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121845 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:51748_
   [junit4]   2> 1121845 T4248 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1121845 T4264 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1121846 T4291 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard2_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1121847 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121847 T4237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1121847 T4248 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121847 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53551",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53551_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121847 T4264 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121847 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:59669_
   [junit4]   2> 1121848 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:52924_
   [junit4]   2> 1121849 T4238 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:53551_
   [junit4]   2> 1121849 T4231 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard3_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1121850 T4291 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1121850 T4277 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard4_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1121849 T4317 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard3_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1121851 T4306 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard4_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1121852 T4291 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121852 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44654",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44654_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121852 T4231 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1121853 T4277 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1121852 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121853 T4306 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1121852 T4317 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1121854 T4231 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121854 T4317 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121854 T4306 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121854 T4277 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1121855 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58991",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58991_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121857 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121857 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54702",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54702_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121858 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121859 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51748",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51748_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121860 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121860 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52924",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52924_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121862 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121862 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53551",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53551_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121863 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121864 T4237 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59669",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59669_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1121865 T4236 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1121968 T4329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4310 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4327 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4296 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4254 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1121969 T4268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1122846 T4305 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 1122847 T4305 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122847 T4305 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122848 T4248 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 1122848 T4264 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 1122848 T4305 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122849 T4248 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122849 T4264 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122849 T4248 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122849 T4305 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122850 T4248 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122849 T4264 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122851 T4264 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122850 T4305 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/'
   [junit4]   2> 1122851 T4248 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122851 T4264 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122852 T4264 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396258807903/testOverseerCol_shard1_replica2/'
   [junit4]   2> 1122852 T4248 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396258806288/testOverseerCol_shard2_replica1/'
   [junit4]   2> 1122852 T4291 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 1122853 T4291 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122854 T4291 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122854 T4291 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122854 T4231 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 1122855 T4317 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 1122855 T4306 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 1122855 T4277 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 1122856 T4317 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122855 T4231 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122855 T4291 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122856 T4231 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122856 T4277 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122856 T4306 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1122857 T4231 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122858 T4306 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122859 T4306 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122857 T4291 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396258810819/testOverseerCol_shard2_replica2/'
   [junit4]   2> 1122859 T4317 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122860 T4306 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122860 T4306 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard4_replica2/'
   [junit4]   2> 1122859 T4277 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1122862 T4277 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122860 T4231 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122863 T4231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396258805825/testOverseerCol_shard3_replica1/'
   [junit4]   2> 1122860 T4317 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1122866 T4317 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122867 T4317 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396258813692/testOverseerCol_shard3_replica2/'
   [junit4]   2> 1122869 T4277 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1122869 T4277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396258809338/testOverseerCol_shard4_replica1/'
   [junit4]   2> 1122874 T4305 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122900 T4264 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122903 T4248 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122907 T4305 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122910 T4264 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122911 T4305 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1122911 T4264 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1122912 T4231 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122917 T4248 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122918 T4264 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 1122918 T4305 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 1122921 T4248 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1122925 T4231 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122926 T4277 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122928 T4231 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1122930 T4248 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 1122949 T4291 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122959 T4277 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122955 T4306 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122966 T4317 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1122992 T4231 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 1122993 T4291 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122995 T4277 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1122996 T4306 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1122995 T4291 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1123003 T4277 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 1123004 T4306 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1123009 T4291 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 1123013 T4306 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 1123003 T4317 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1123055 T4317 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1123087 T4317 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 1123189 T4305 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1123235 T4305 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1123238 T4305 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1123275 T4248 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1123281 T4305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1123284 T4305 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1123286 T4264 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1123287 T4305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1123289 T4305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1123289 T4305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1123290 T4305 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1123290 T4248 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1123292 T4264 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1123293 T4248 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1123284 T4277 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1123294 T4264 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1123291 T4305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1123295 T4305 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1123296 T4305 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using configuration from collection testOverseerCol
   [junit4]   2> 1123296 T4305 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1123297 T4305 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/, dataDir=./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/data/
   [junit4]   2> 1123297 T4305 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17199b7
   [junit4]   2> 1123299 T4305 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/data
   [junit4]   2> 1123299 T4305 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/data/index/
   [junit4]   2> 1123299 T4305 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica1] Solr index directory './solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1123300 T4305 oasc.CachingDirectoryFactory.get return new directory for ./solrtest-OverseerRolesTest-1396258804408/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396258812280/testOverseerCol_shard1_replica1/data/index
   [junit4]   2> 1123301 T4305 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=38.1494140625, floorSegmentMB=0.2451171875, forceMergeDeletesPctAllowed=17.14289206444731, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4926635882590803
   [junit4]   2> 1123302 T4305 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@123d808 lockFactory=org.apache.lucene.store.SingleInstanceLockF

[...truncated too long message...]

ption$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
   [junit4]   2> 	... 19 more
   [junit4]   2> 
   [junit4]   2> 1141742 T4391 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1141742 T4391 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerRolesTest -Dtests.method=testDistribSearch -Dtests.seed=725FF5CB1B30450C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=no_NO_NY -Dtests.timezone=Asia/Taipei -Dtests.file.encoding=UTF-8
   [junit4] ERROR   30.8s J1 | OverseerRolesTest.testDistribSearch <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4393, name=Overseer-91505217006010374-127.0.0.1:44654_-n_0000000011, state=RUNNABLE, group=Overseer collection creation process.]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([725FF5CB1B30450C:F3B97BD36C6F2530]:0)
   [junit4]    > Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([725FF5CB1B30450C]:0)
   [junit4]    > 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:275)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000006
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:276)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:273)
   [junit4]    > 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:206)
   [junit4]    > 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:258)
   [junit4]    > 	... 1 more
   [junit4]   2> 1141744 T4214 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=267), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=267)), sim=DefaultSimilarity, locale=no_NO_NY, timezone=Asia/Taipei
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=231853728,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SuggesterTSTTest, SolrIndexConfigTest, TestIndexingPerformance, TestLazyCores, BlockCacheTest, CachingDirectoryFactoryTest, TestHashPartitioner, FileUtilsTest, TestCollationFieldDocValues, StressHdfsTest, TestFastOutputStream, TestOmitPositions, TestReloadAndDeleteDocs, RequiredFieldsTest, XsltUpdateRequestHandlerTest, TestPartialUpdateDeduplication, StandardRequestHandlerTest, DebugComponentTest, TestRangeQuery, TestReload, ConvertedLegacyTest, TestOverriddenPrefixQueryForCustomFieldType, TestUtils, DistribCursorPagingTest, TestFoldingMultitermQuery, MigrateRouteKeyTest, TestManagedStopFilterFactory, StatsComponentTest, TestNonNRTOpen, TestQuerySenderNoQuery, ZkSolrClientTest, BasicDistributedZk2Test, DeleteShardTest, TestSolrDeletionPolicy2, NotRequiredUniqueKeyTest, TestFastLRUCache, TestIndexSearcher, BlockDirectoryTest, TestSchemaSimilarityResource, SolrPluginUtilsTest, TestCoreDiscovery, SpellCheckComponentTest, HdfsChaosMonkeySafeLeaderTest, HighlighterConfigTest, SuggesterWFSTTest, TestSchemaResource, NumericFieldsTest, TestDFRSimilarityFactory, TestStressReorder, CursorPagingTest, SignatureUpdateProcessorFactoryTest, ZkNodePropsTest, TestDistributedGrouping, AnalysisErrorHandlingTest, TestFieldResource, PrimUtilsTest, TestFastWriter, OverseerStatusTest, TestSolrQueryParserDefaultOperatorResource, DocumentAnalysisRequestHandlerTest, TestFreeTextSuggestions, TestWordDelimiterFilterFactory, TestMultiCoreConfBootstrap, HighlighterTest, TestAddFieldRealTimeGet, TestStressLucene, OpenExchangeRatesOrgProviderTest, TestInitQParser, CurrencyFieldOpenExchangeTest, DistributedSuggestComponentTest, TestPhraseSuggestions, DocValuesMultiTest, TestAtomicUpdateErrorCases, SolrCmdDistributorTest, EnumFieldTest, TestStressRecovery, TestStandardQParsers, TestRandomFaceting, TestFieldTypeCollectionResource, TestRTGBase, RequestHandlersTest, ConnectionManagerTest, PingRequestHandlerTest, SolrRequestParserTest, AliasIntegrationTest, DateFieldTest, CoreMergeIndexesAdminHandlerTest, TestRandomMergePolicy, ModifyConfFileTest, ShardRoutingCustomTest, DirectUpdateHandlerTest, DeleteInactiveReplicaTest, DefaultValueUpdateProcessorTest, TestComponentsName, TestAnalyzedSuggestions, TestModifyConfFiles, DisMaxRequestHandlerTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, ClusterStateUpdateTest, TestZkChroot, ZkCLITest, AutoCommitTest, TermVectorComponentDistributedTest, TestStressVersions, SimpleFacetsTest, SolrCoreTest, TestGroupingSearch, TestFiltering, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, TestArbitraryIndexDir, LoggingHandlerTest, TestCollationField, TestCSVResponseWriter, QueryParsingTest, CSVRequestHandlerTest, TestBinaryResponseWriter, AlternateDirectoryTest, MultiTermTest, SampleTest, TestBinaryField, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, TestSolrCoreProperties, TestLuceneMatchVersion, TestCharFilters, TestXIncludeConfig, TestNumberUtils, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, ScriptEngineTest, PreAnalyzedFieldTest, TestSystemIdResolver, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, AbstractAnalyticsStatsTest, NoFacetTest, ExpressionTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, QueryFacetTest, RangeFacetTest, FunctionTest, AsyncMigrateRouteKeyTest, DeleteReplicaTest, OverseerRolesTest]
   [junit4] Completed on J1 in 31.30s, 1 test, 1 error <<< FAILURES!

[...truncated 168 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 394 suites, 1653 tests, 1 error, 49 ignored (28 assumptions)

Total time: 56 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure