You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/30 03:04:21 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0) - Build # 9836 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9836/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=6446, name=Overseer-91497506466824204-127.0.0.1:36150_q_%2Fl-n_0000000010, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6446, name=Overseer-91497506466824204-127.0.0.1:36150_q_%2Fl-n_0000000010, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([21F223FB489DD484:A014ADE33FC2B4B8]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([21F223FB489DD484]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:280)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000006
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:176)
	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:173)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:207)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:263)
	... 1 more




Build Log:
[...truncated 11640 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917
   [junit4]   2> 1006052 T6282 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /q_/l
   [junit4]   2> 1006055 T6282 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1006055 T6282 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1006056 T6283 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1006156 T6282 oasc.ZkTestServer.run start zk server on port:36282
   [junit4]   2> 1006157 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014195 T6286 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 8037ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 1014196 T6289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20092d42 name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014196 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014197 T6282 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1014200 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014201 T6291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@212352d name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014201 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014201 T6282 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1014202 T6282 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1014204 T6282 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1014204 T6282 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1014205 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1014206 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1014208 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1014208 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1014210 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1014210 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1014211 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1014212 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1014213 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1014213 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1014214 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1014214 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1014216 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1014216 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1014218 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1014218 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1014219 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1014220 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1014221 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1014222 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1014223 T6282 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1014223 T6282 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1014451 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1014467 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56136
   [junit4]   2> 1014468 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1014468 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1014468 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090
   [junit4]   2> 1014469 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090/'
   [junit4]   2> 1014484 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090/solr.xml
   [junit4]   2> 1014495 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090
   [junit4]   2> 1014496 T6282 oasc.CoreContainer.<init> New CoreContainer 1881442551
   [junit4]   2> 1014496 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090/]
   [junit4]   2> 1014497 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1014497 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1014497 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1014498 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1014498 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1014498 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1014499 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1014499 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1014499 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1014500 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1014500 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1014501 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1014501 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1014501 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1014502 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014503 T6302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@405979c6 name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014503 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014505 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014505 T6304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@144334b9 name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014506 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014506 T6282 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1014508 T6282 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1014510 T6282 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1014512 T6282 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1014513 T6282 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1014521 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1014522 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56136_q_%2Fl
   [junit4]   2> 1014522 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56136_q_%2Fl
   [junit4]   2> 1014524 T6282 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1014525 T6282 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1014526 T6282 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:56136_q_%2Fl
   [junit4]   2> 1014527 T6282 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1014528 T6282 oasc.Overseer.start Overseer (id=91497506466824195-127.0.0.1:56136_q_%2Fl-n_0000000000) starting
   [junit4]   2> 1014529 T6282 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1014533 T6306 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1014534 T6282 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1014534 T6306 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 1014536 T6282 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1014537 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1014539 T6305 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1014540 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090
   [junit4]   2> 1014545 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090/conf/
   [junit4]   2> 1014637 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1014638 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1014638 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1014639 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014641 T6309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@664c9a9b name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014641 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014642 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1014643 T6282 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1014882 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1014883 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1014884 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50310
   [junit4]   2> 1014885 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1014885 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1014886 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508
   [junit4]   2> 1014886 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/'
   [junit4]   2> 1014901 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/solr.xml
   [junit4]   2> 1014912 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508
   [junit4]   2> 1014912 T6282 oasc.CoreContainer.<init> New CoreContainer 2004043948
   [junit4]   2> 1014912 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/]
   [junit4]   2> 1014913 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1014913 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1014913 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1014913 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1014913 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1014914 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1014914 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1014914 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1014914 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1014915 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1014915 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1014915 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1014915 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1014916 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1014916 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014917 T6320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@265f5d94 name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014917 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014919 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1014920 T6322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d823518 name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1014920 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1014923 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1015925 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50310_q_%2Fl
   [junit4]   2> 1015925 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50310_q_%2Fl
   [junit4]   2> 1015927 T6304 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1015927 T6309 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1015927 T6322 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1015929 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508
   [junit4]   2> 1015935 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/conf/
   [junit4]   2> 1016053 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1016054 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1016054 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1016282 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1016282 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1016283 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46445
   [junit4]   2> 1016284 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1016284 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1016284 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920
   [junit4]   2> 1016284 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920/'
   [junit4]   2> 1016294 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920/solr.xml
   [junit4]   2> 1016302 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920
   [junit4]   2> 1016302 T6282 oasc.CoreContainer.<init> New CoreContainer 1184530911
   [junit4]   2> 1016302 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920/]
   [junit4]   2> 1016303 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1016303 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1016303 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1016303 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1016303 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1016304 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1016304 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1016304 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1016304 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1016305 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1016305 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1016305 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1016305 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1016306 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1016306 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1016308 T6334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@135ff32b name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1016308 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1016309 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1016310 T6336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@362fb5ef name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1016310 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1016314 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1017315 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46445_q_%2Fl
   [junit4]   2> 1017316 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46445_q_%2Fl
   [junit4]   2> 1017320 T6322 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017320 T6336 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017320 T6309 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017323 T6304 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017325 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920
   [junit4]   2> 1017334 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920/conf/
   [junit4]   2> 1017428 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1017429 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1017429 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1017768 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1017769 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1017773 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54481
   [junit4]   2> 1017774 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1017775 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1017775 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295
   [junit4]   2> 1017775 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/'
   [junit4]   2> 1017791 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/solr.xml
   [junit4]   2> 1017802 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295
   [junit4]   2> 1017803 T6282 oasc.CoreContainer.<init> New CoreContainer 1110734044
   [junit4]   2> 1017803 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/]
   [junit4]   2> 1017804 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1017804 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1017804 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1017805 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1017805 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1017805 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1017806 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1017806 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1017806 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1017807 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1017807 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1017808 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1017808 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1017808 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1017819 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1017820 T6348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a83945e name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1017820 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1017821 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1017822 T6350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@746ec0d5 name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1017822 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1017826 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1018828 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54481_q_%2Fl
   [junit4]   2> 1018829 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54481_q_%2Fl
   [junit4]   2> 1018831 T6322 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1018831 T6350 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1018832 T6309 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1018831 T6336 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1018831 T6304 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1018833 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295
   [junit4]   2> 1018839 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/conf/
   [junit4]   2> 1018966 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1018968 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1018968 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1019197 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1019198 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1019199 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36150
   [junit4]   2> 1019199 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1019200 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1019200 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833
   [junit4]   2> 1019200 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833/'
   [junit4]   2> 1019210 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833/solr.xml
   [junit4]   2> 1019219 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833
   [junit4]   2> 1019219 T6282 oasc.CoreContainer.<init> New CoreContainer 1115299270
   [junit4]   2> 1019219 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833/]
   [junit4]   2> 1019220 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1019220 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1019220 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1019220 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1019221 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1019221 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1019221 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1019221 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1019221 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1019222 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1019222 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1019222 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1019222 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1019223 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1019223 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1019224 T6362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e3236e name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1019225 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1019226 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1019227 T6364 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2659fbe name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1019227 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1019231 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1020232 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36150_q_%2Fl
   [junit4]   2> 1020234 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36150_q_%2Fl
   [junit4]   2> 1020236 T6304 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020236 T6336 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020236 T6364 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020236 T6322 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020236 T6350 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020236 T6309 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1020239 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833
   [junit4]   2> 1020245 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833/conf/
   [junit4]   2> 1020374 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1020375 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1020375 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1020608 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 1020608 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1020610 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40476
   [junit4]   2> 1020610 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1020610 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1020611 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240
   [junit4]   2> 1020611 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240/'
   [junit4]   2> 1020622 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240/solr.xml
   [junit4]   2> 1020629 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240
   [junit4]   2> 1020629 T6282 oasc.CoreContainer.<init> New CoreContainer 308798451
   [junit4]   2> 1020630 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240/]
   [junit4]   2> 1020630 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1020630 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1020631 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1020631 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1020631 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1020631 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1020632 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1020632 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1020632 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1020632 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1020633 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1020633 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1020633 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1020633 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1020634 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1020705 T6376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d670a53 name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1020705 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1020743 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1020743 T6378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@219a8ef8 name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1020743 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1020747 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1021749 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40476_q_%2Fl
   [junit4]   2> 1021750 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40476_q_%2Fl
   [junit4]   2> 1021751 T6336 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6364 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6309 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6304 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6322 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6350 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021751 T6378 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1021754 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240
   [junit4]   2> 1021762 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240/conf/
   [junit4]   2> 1021934 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1021935 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1021935 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1022326 T6282 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 1022327 T6282 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1022332 T6282 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36840
   [junit4]   2> 1022333 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1022333 T6282 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1022333 T6282 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801
   [junit4]   2> 1022334 T6282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801/'
   [junit4]   2> 1022350 T6282 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801/solr.xml
   [junit4]   2> 1022382 T6282 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801
   [junit4]   2> 1022383 T6282 oasc.CoreContainer.<init> New CoreContainer 1319892244
   [junit4]   2> 1022384 T6282 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801/]
   [junit4]   2> 1022384 T6282 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1022385 T6282 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1022385 T6282 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1022385 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1022386 T6282 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1022386 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1022386 T6282 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1022387 T6282 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1022387 T6282 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1022388 T6282 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1022388 T6282 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1022388 T6282 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1022389 T6282 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36282/solr
   [junit4]   2> 1022389 T6282 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1022391 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1022392 T6390 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2841acd5 name:ZooKeeperConnection Watcher:127.0.0.1:36282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1022392 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1022394 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1022396 T6392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c93fcbb name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1022396 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1022402 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1023406 T6282 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36840_q_%2Fl
   [junit4]   2> 1023410 T6282 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36840_q_%2Fl
   [junit4]   2> 1023414 T6336 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6304 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6322 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6364 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6378 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6309 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6392 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023415 T6350 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1023418 T6282 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801
   [junit4]   2> 1023427 T6282 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801/conf/
   [junit4]   2> 1023595 T6282 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1023596 T6282 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1023596 T6282 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1023597 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1023598 T6395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a652961 name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1023598 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1023599 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1023601 T6282 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1023602 T6397 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bf7766c name:ZooKeeperConnection Watcher:127.0.0.1:36282/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1023602 T6282 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1023603 T6282 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1023607 T6297 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=testOverseerCol&action=CREATE&numShards=4&wt=javabin&version=2
   [junit4]   2> 1023609 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023609 T6306 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1023610 T6306 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1023610 T6306 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1023610 T6306 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 1023610 T6306 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 1023612 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023613 T6305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1023613 T6305 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 1023614 T6305 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 1023615 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023615 T6378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6364 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023616 T6397 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6350 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023615 T6336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023713 T6306 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2, shard3, shard4] , replicationFactor : 2
   [junit4]   2> 1023713 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:54481_q_%2Fl
   [junit4]   2> 1023714 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:50310_q_%2Fl
   [junit4]   2> 1023714 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:56136_q_%2Fl
   [junit4]   2> 1023715 T6346 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard1_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1023716 T6346 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1023717 T6346 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023717 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023717 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:46445_q_%2Fl
   [junit4]   2> 1023718 T6318 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard1_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1023721 T6318 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1023721 T6318 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023732 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:40476_q_%2Fl
   [junit4]   2> 1023732 T6299 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard2_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1023732 T6329 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard2_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1023733 T6305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1023734 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54481/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54481_q_%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023735 T6299 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1023735 T6329 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1023736 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023736 T6329 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023737 T6299 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023737 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50310/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50310_q_%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023741 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023741 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56136/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56136_q_%2Fl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023744 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023745 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46445/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46445_q_%2Fl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023748 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023748 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:36840_q_%2Fl
   [junit4]   2> 1023749 T6371 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard3_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1023750 T6371 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1023751 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023751 T6371 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023751 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40476/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40476_q_%2Fl",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023753 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023759 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:36150_q_%2Fl
   [junit4]   2> 1023760 T6306 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:54481_q_%2Fl
   [junit4]   2> 1023763 T6358 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard4_replica1&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1023763 T6343 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard4_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard4&wt=javabin&version=2
   [junit4]   2> 1023764 T6387 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=testOverseerCol_shard3_replica2&action=CREATE&numShards=4&collection=testOverseerCol&shard=shard3&wt=javabin&version=2
   [junit4]   2> 1023765 T6343 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1023765 T6358 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down collection=testOverseerCol
   [junit4]   2> 1023766 T6387 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down collection=testOverseerCol
   [junit4]   2> 1023766 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023766 T6343 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023766 T6358 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023767 T6387 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023767 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023767 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54481/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54481_q_%2Fl",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023772 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36150/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36150_q_%2Fl",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023773 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023774 T6305 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36840/q_/l",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36840_q_%2Fl",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023776 T6304 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023878 T6364 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6304 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6397 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6350 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1023878 T6322 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 1024717 T6346 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 1024718 T6346 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024719 T6346 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024719 T6346 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024720 T6346 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024720 T6346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/testOverseerCol_shard1_replica1/'
   [junit4]   2> 1024722 T6318 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 1024723 T6318 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024723 T6318 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024723 T6318 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024724 T6318 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024724 T6318 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/'
   [junit4]   2> 1024737 T6329 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 1024737 T6299 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 1024738 T6329 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024738 T6299 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024739 T6329 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024739 T6329 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024739 T6299 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024739 T6299 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024741 T6329 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024741 T6329 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty2-1396141160920/testOverseerCol_shard2_replica2/'
   [junit4]   2> 1024743 T6299 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024743 T6299 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-controljetty-1396141159090/testOverseerCol_shard2_replica1/'
   [junit4]   2> 1024751 T6371 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 1024752 T6346 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024752 T6371 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024753 T6371 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024753 T6371 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024754 T6318 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024754 T6371 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024755 T6371 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty5-1396141165240/testOverseerCol_shard3_replica1/'
   [junit4]   2> 1024766 T6346 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024767 T6358 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 1024767 T6387 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 1024768 T6387 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024766 T6343 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 1024768 T6358 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024769 T6343 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 1024769 T6318 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024770 T6358 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024770 T6358 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024770 T6346 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024770 T6387 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024771 T6358 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024771 T6358 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty4-1396141163833/testOverseerCol_shard4_replica1/'
   [junit4]   2> 1024770 T6343 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024771 T6318 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024771 T6387 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024770 T6329 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024774 T6299 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024772 T6343 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 1024782 T6387 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024782 T6387 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty6-1396141166801/testOverseerCol_shard3_replica2/'
   [junit4]   2> 1024783 T6343 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024783 T6343 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty3-1396141162295/testOverseerCol_shard4_replica2/'
   [junit4]   2> 1024787 T6318 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 1024789 T6371 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024800 T6299 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024803 T6299 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024805 T6346 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 1024805 T6371 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024806 T6329 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024807 T6371 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024809 T6329 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024811 T6299 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 1024831 T6371 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 1024835 T6358 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024833 T6329 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 1024881 T6387 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024918 T6343 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 1024933 T6387 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024922 T6358 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024936 T6387 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024936 T6358 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024958 T6387 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 1024958 T6358 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 1024995 T6343 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1025086 T6343 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1025113 T6343 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 1025224 T6318 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025225 T6318 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025226 T6318 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1025233 T6318 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1025233 T6346 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025249 T6346 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025250 T6346 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1025250 T6318 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1025267 T6346 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1025269 T6318 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1025271 T6346 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1025301 T6299 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025272 T6318 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1025304 T6299 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1025305 T6299 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1025307 T6318 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1025311 T6318 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1025311 T6318 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1025312 T6318 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1025313 T6299 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1025313 T6318 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1025314 T6318 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1025314 T6318 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using configuration from collection testOverseerCol
   [junit4]   2> 1025314 T6318 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1025315 T6318 oasc.SolrCore.<init> [testOverseerCol_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/data/
   [junit4]   2> 1025315 T6318 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7905633a
   [junit4]   2> 1025315 T6299 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1025316 T6346 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1025316 T6318 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/data
   [junit4]   2> 1025317 T6318 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/data/index/
   [junit4]   2> 1025317 T6318 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1025318 T6318 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerRolesTest-1396141150917/org.apache.solr.cloud.OverseerRolesTest-jetty1-1396141159508/testOverseerCol_shard1_replica2/data/index
   [junit4]   2> 1025318 T6318 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=33.859375, floorSegmentMB=0.6162109375, forceMergeDele

[...truncated too long message...]

common.cloud.SolrZkClient.delete(SolrZkClient.java:173)
   [junit4]    > 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:207)
   [junit4]    > 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:263)
   [junit4]    > 	... 1 more
   [junit4]   2> 1040216 T6282 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 34734 T6281 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> 1041163 T6460 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1041248 T6364 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1041248 T6364 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1041332 T6378 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1041332 T6378 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1041332 T6378 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1041333 T6378 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1041417 T6392 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1041418 T6392 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=sv_SE, timezone=Indian/Maldives
   [junit4]   2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=1,free=89572736,total=518520832
   [junit4]   2> NOTE: All tests run in this JVM: [TestAnalyzeInfixSuggestions, TestQuerySenderNoQuery, AnalysisErrorHandlingTest, SignatureUpdateProcessorFactoryTest, TestCloudManagedSchema, TestFieldTypeCollectionResource, ConvertedLegacyTest, HdfsLockFactoryTest, MultiTermTest, TestJmxIntegration, TestFastOutputStream, CoreAdminRequestStatusTest, TestConfigSets, TestSolrXmlPersistence, TestStressLucene, TestDynamicFieldCollectionResource, TestUpdate, HdfsDirectoryTest, TestStressRecovery, HighlighterConfigTest, TestNonDefinedSimilarityFactory, ZkNodePropsTest, NumericFieldsTest, TestSchemaVersionResource, TestDistributedSearch, BasicDistributedZk2Test, NoCacheHeaderTest, TestDistribDocBasedVersion, TestCollationField, TimeZoneUtilsTest, PluginInfoTest, TestSolrQueryParserResource, CachingDirectoryFactoryTest, TestSolrQueryParserDefaultOperatorResource, FullSolrCloudDistribCmdsTest, OutputWriterTest, SolrCmdDistributorTest, DistributedQueryComponentCustomSortTest, TestSerializedLuceneMatchVersion, TestDocBasedVersionConstraints, TestSolrDeletionPolicy2, TestClassNameShortening, TestArbitraryIndexDir, TestCSVResponseWriter, MigrateRouteKeyTest, StandardRequestHandlerTest, HighlighterTest, TestRandomFaceting, TestQuerySenderListener, TestSolr4Spatial, TestCollectionAPI, IndexSchemaRuntimeFieldTest, ParsingFieldUpdateProcessorsTest, TestUtils, BlockDirectoryTest, TestImplicitCoreProperties, TestFileDictionaryLookup, ChaosMonkeySafeLeaderTest, DistributedDebugComponentTest, ChaosMonkeyNothingIsSafeTest, RequestHandlersTest, TestHighFrequencyDictionaryFactory, TestDefaultSearchFieldResource, TestSweetSpotSimilarityFactory, SuggestComponentTest, TestSimpleQParserPlugin, FileUtilsTest, CursorMarkTest, TestStandardQParsers, BadIndexSchemaTest, RequiredFieldsTest, SuggesterWFSTTest, CollectionsAPIAsyncDistributedZkTest, PathHierarchyTokenizerFactoryTest, TestLazyCores, RAMDirectoryFactoryTest, DefaultValueUpdateProcessorTest, TestPhraseSuggestions, SolrRequestParserTest, TestCollapseQParserPlugin, TestNumberUtils, TestWordDelimiterFilterFactory, DistributedSuggestComponentTest, CurrencyFieldOpenExchangeTest, SolrCoreCheckLockOnStartupTest, SimplePostToolTest, DistanceFunctionTest, TestCSVLoader, HdfsRecoveryZkTest, DocumentAnalysisRequestHandlerTest, TestXIncludeConfig, TestAnalyzedSuggestions, TestCoreDiscovery, ShardRoutingCustomTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, TestReplicationHandler, LeaderElectionTest, ZkSolrClientTest, TestRandomDVFaceting, TermVectorComponentDistributedTest, TestStressReorder, TestMultiCoreConfBootstrap, TestStressVersions, TestRangeQuery, SpellCheckComponentTest, TestFunctionQuery, BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SpellCheckCollatorTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, SuggesterTSTTest, PolyFieldTest, SchemaVersionSpecificBehaviorTest, TestAtomicUpdateErrorCases, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, DebugComponentTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, TestOmitPositions, XmlUpdateRequestHandlerTest, TestValueSourceCache, FieldAnalysisRequestHandlerTest, SolrPluginUtilsTest, ReturnFieldsTest, JSONWriterTest, QueryParsingTest, PingRequestHandlerTest, TestComponentsName, CoreContainerCoreInitFailuresTest, TestLFUCache, ResponseLogComponentTest, BadComponentTest, SampleTest, MinimalSchemaTest, TestFuzzyAnalyzedSuggestions, DirectSolrConnectionTest, TestCodecSupport, SynonymTokenizerTest, TestIBSimilarityFactory, TestDefaultSimilarityFactory, TestJmxMonitoredMap, TestFastLRUCache, PreAnalyzedFieldTest, DateFieldTest, DOMUtilTest, SliceStateTest, SystemInfoHandlerTest, TestRTGBase, SolrTestCaseJ4Test, TestDistributedMissingSort, TestHighlightDedupGrouping, TestCollationKeyRangeQueries, AliasIntegrationTest, ConnectionManagerTest, CustomCollectionTest, OverseerRolesTest]
   [junit4] Completed on J1 in 35.50s, 1 test, 1 error <<< FAILURES!

[...truncated 161 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 380 suites, 1605 tests, 1 error, 37 ignored (16 assumptions)

Total time: 54 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure