You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/05 04:16:06 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b02) - Build # 8964 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8964/
Java: 32bit/jdk1.7.0_60-ea-b02 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.UnloadDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest:     1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([DAF1886D0F823A54]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.UnloadDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([DAF1886D0F823A54]:0)




Build Log:
[...truncated 10498 lines...]
   [junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
   [junit4]   2> 634358 T3671 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 634364 T3671 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490
   [junit4]   2> 634364 T3671 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 634365 T3672 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 634465 T3671 oasc.ZkTestServer.run start zk server on port:51855
   [junit4]   2> 634466 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 634713 T3679 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14bb789 name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 634713 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 634713 T3671 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 634718 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 634719 T3682 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c08f0 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 634719 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 634720 T3671 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 634722 T3671 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 634724 T3671 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 634726 T3671 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 634730 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 634731 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 634749 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 634749 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 634753 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 634753 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 634755 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 634756 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 634759 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 634759 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 634761 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 634762 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 634764 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 634765 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 634774 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 634774 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 634789 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 634790 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 634792 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 634793 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 634795 T3671 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 634795 T3671 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 634944 T3671 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 634948 T3671 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37736
   [junit4]   2> 634948 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 634948 T3671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 634949 T3671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926
   [junit4]   2> 634949 T3671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926/'
   [junit4]   2> 634961 T3671 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926/solr.xml
   [junit4]   2> 634989 T3671 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 634990 T3671 oasc.CoreContainer.<init> New CoreContainer 15982569
   [junit4]   2> 634990 T3671 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926/]
   [junit4]   2> 634991 T3671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 634991 T3671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 634992 T3671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 634992 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 634992 T3671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 634992 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 634993 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 634993 T3671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 634993 T3671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 634995 T3671 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 634996 T3671 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 634996 T3671 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 634996 T3671 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51855/solr
   [junit4]   2> 634996 T3671 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 635001 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 635002 T3694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@150a2de name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 635003 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 635007 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 635008 T3697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97ebe2 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 635008 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 635010 T3671 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 635015 T3671 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 635022 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 635024 T3671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37736_
   [junit4]   2> 635025 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37736_
   [junit4]   2> 635027 T3671 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 635029 T3671 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 635033 T3671 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 635035 T3671 oasc.Overseer.start Overseer (id=91022396784115715-127.0.0.1:37736_-n_0000000000) starting
   [junit4]   2> 635039 T3671 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 635042 T3699 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 635043 T3671 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 635045 T3671 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 635046 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 635052 T3698 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 635054 T3671 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926
   [junit4]   2> 635055 T3671 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388891552926/conf/
   [junit4]   2> 635086 T3671 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 635087 T3671 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 635087 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 635088 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 635091 T3703 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e2b7a1 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 635091 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 635092 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 635094 T3671 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 635178 T3671 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 635180 T3671 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49191
   [junit4]   2> 635181 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 635181 T3671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 635181 T3671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221
   [junit4]   2> 635182 T3671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/'
   [junit4]   2> 635197 T3671 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/solr.xml
   [junit4]   2> 635228 T3671 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 635228 T3671 oasc.CoreContainer.<init> New CoreContainer 24660981
   [junit4]   2> 635229 T3671 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/]
   [junit4]   2> 635230 T3671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 635230 T3671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 635231 T3671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 635231 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 635231 T3671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 635232 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 635232 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 635233 T3671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 635233 T3671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 635236 T3671 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 635237 T3671 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 635237 T3671 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 635238 T3671 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51855/solr
   [junit4]   2> 635238 T3671 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 635246 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 635247 T3715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dfa068 name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 635252 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 635265 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 635268 T3718 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188fdc7 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 635268 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 635277 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 636284 T3671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49191_
   [junit4]   2> 636285 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49191_
   [junit4]   2> 636289 T3697 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 636290 T3703 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 636290 T3718 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 636301 T3671 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221
   [junit4]   2> 636303 T3671 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/conf/
   [junit4]   2> 636341 T3671 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 636343 T3671 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 636343 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 636475 T3671 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 636478 T3671 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38768
   [junit4]   2> 636479 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 636480 T3671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 636480 T3671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470
   [junit4]   2> 636481 T3671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/'
   [junit4]   2> 636500 T3671 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/solr.xml
   [junit4]   2> 636548 T3671 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 636549 T3671 oasc.CoreContainer.<init> New CoreContainer 14527773
   [junit4]   2> 636549 T3671 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/]
   [junit4]   2> 636550 T3671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 636550 T3671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 636551 T3671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 636551 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 636551 T3671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 636552 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 636552 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 636552 T3671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 636552 T3671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 636555 T3671 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 636556 T3671 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 636556 T3671 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 636557 T3671 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51855/solr
   [junit4]   2> 636557 T3671 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 636574 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 636589 T3731 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e4c5a9 name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 636590 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 636593 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 636594 T3734 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10fa02c name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 636595 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 636599 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 637609 T3671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38768_
   [junit4]   2> 637610 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38768_
   [junit4]   2> 637614 T3734 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 637614 T3703 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 637614 T3718 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 637614 T3697 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 637618 T3671 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470
   [junit4]   2> 637619 T3671 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/conf/
   [junit4]   2> 637646 T3671 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 637647 T3671 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 637647 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 637732 T3671 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 637734 T3671 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60336
   [junit4]   2> 637735 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 637736 T3671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 637736 T3671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774
   [junit4]   2> 637736 T3671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/'
   [junit4]   2> 637748 T3671 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/solr.xml
   [junit4]   2> 637773 T3671 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 637774 T3671 oasc.CoreContainer.<init> New CoreContainer 27869045
   [junit4]   2> 637774 T3671 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/]
   [junit4]   2> 637775 T3671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 637775 T3671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 637776 T3671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 637776 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 637776 T3671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 637777 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 637777 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 637777 T3671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 637778 T3671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 637780 T3671 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 637780 T3671 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 637781 T3671 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 637781 T3671 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51855/solr
   [junit4]   2> 637781 T3671 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 637782 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637784 T3747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bac84d name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637784 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637786 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 637787 T3750 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17aed40 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 637788 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 637791 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 638795 T3671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60336_
   [junit4]   2> 638798 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60336_
   [junit4]   2> 638802 T3718 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 638802 T3703 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 638802 T3750 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 638802 T3697 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 638802 T3734 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 638807 T3671 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774
   [junit4]   2> 638808 T3671 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/conf/
   [junit4]   2> 638837 T3671 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 638837 T3671 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 638838 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 638943 T3671 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 638954 T3671 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36103
   [junit4]   2> 638955 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 638955 T3671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 638956 T3671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964
   [junit4]   2> 638956 T3671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964/'
   [junit4]   2> 638970 T3671 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964/solr.xml
   [junit4]   2> 638997 T3671 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 638997 T3671 oasc.CoreContainer.<init> New CoreContainer 27846
   [junit4]   2> 638998 T3671 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964/]
   [junit4]   2> 638999 T3671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 638999 T3671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 639000 T3671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 639000 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 639000 T3671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 639001 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 639001 T3671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 639002 T3671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 639002 T3671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 639005 T3671 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 639005 T3671 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 639005 T3671 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 639006 T3671 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51855/solr
   [junit4]   2> 639006 T3671 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 639009 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 639012 T3764 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23663b name:ZooKeeperConnection Watcher:127.0.0.1:51855 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 639014 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 639020 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 639022 T3767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c663e5 name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 639022 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 639027 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 640030 T3671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36103_
   [junit4]   2> 640031 T3671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36103_
   [junit4]   2> 640034 T3734 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640034 T3750 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640034 T3703 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640035 T3718 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640036 T3767 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640035 T3697 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 640041 T3671 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964
   [junit4]   2> 640042 T3671 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388891556964/conf/
   [junit4]   2> 640075 T3671 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 640076 T3671 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 640076 T3671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 640078 T3711 oasc.ZkController.publish publishing core=unloadcollection1 state=down
   [junit4]   2> 640079 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 640079 T3711 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 640081 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 640082 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49191",
   [junit4]   2> 	  "core":"unloadcollection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49191_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 640082 T3698 oasc.Overseer$ClusterStateUpdater.createCollection Create collection unloadcollection with shards [shard1]
   [junit4]   2> 640082 T3698 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 640085 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 640086 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 640086 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 640086 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 640086 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 640087 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 640088 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641079 T3711 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection1
   [junit4]   2> 641080 T3711 oasc.CoreContainer.create Creating SolrCore 'unloadcollection1' using instanceDir: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/unloadcollection1
   [junit4]   2> 641081 T3711 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 641081 T3711 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:unloadcollection
   [junit4]   2> 641082 T3711 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 641083 T3711 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 641083 T3711 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection
   [junit4]   2> 641087 T3711 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 641088 T3711 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/unloadcollection1/'
   [junit4]   2> 641123 T3711 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 641152 T3711 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 641155 T3711 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 641164 T3711 oass.IndexSchema.readSchema [unloadcollection1] Schema name=test
   [junit4]   2> 641464 T3711 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 641471 T3711 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 641474 T3711 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 641498 T3711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 641501 T3711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 641505 T3711 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 641507 T3711 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 641507 T3711 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 641508 T3711 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 641512 T3711 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 641513 T3711 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 641513 T3711 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 641513 T3711 oasc.SolrCore.<init> [unloadcollection1] Opening new SolrCore at ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388891553221/unloadcollection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n/
   [junit4]   2> 641514 T3711 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17b24c
   [junit4]   2> 641516 T3711 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n
   [junit4]   2> 641516 T3711 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n/index/
   [junit4]   2> 641517 T3711 oasc.SolrCore.initIndex WARN [unloadcollection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n/index' doesn't exist. Creating new index...
   [junit4]   2> 641517 T3711 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n/index
   [junit4]   2> 641518 T3711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=37.3896484375, floorSegmentMB=1.4375, forceMergeDeletesPctAllowed=13.593839988159031, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 641519 T3711 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7ad875 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@141d93f),segFN=segments_1,generation=1}
   [junit4]   2> 641519 T3711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 641524 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 641525 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 641525 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 641526 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 641526 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 641527 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 641527 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 641528 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 641528 T3711 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 641529 T3711 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 641530 T3711 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 641530 T3711 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 641531 T3711 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 641532 T3711 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 641532 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 641533 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 641534 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 641539 T3711 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 641544 T3711 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 641545 T3711 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 641546 T3711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=85.58203125, floorSegmentMB=0.53515625, forceMergeDeletesPctAllowed=20.43970905844027, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.17476800285564592
   [junit4]   2> 641547 T3711 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7ad875 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@141d93f),segFN=segments_1,generation=1}
   [junit4]   2> 641547 T3711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 641547 T3711 oass.SolrIndexSearcher.<init> Opening Searcher@1fc6e75 main
   [junit4]   2> 641551 T3769 oasc.SolrCore.registerSearcher [unloadcollection1] Registered new searcher Searcher@1fc6e75 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 641553 T3711 oasc.CoreContainer.registerCore registering core: unloadcollection1
   [junit4]   2> 641553 T3711 oasc.ZkController.register Register replica - core:unloadcollection1 address:http://127.0.0.1:49191 collection:unloadcollection shard:shard1
   [junit4]   2> 641555 T3711 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leader_elect/shard1/election
   [junit4]   2> 641559 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 641561 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641561 T3711 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 641562 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 641562 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1848 name=unloadcollection1 org.apache.solr.core.SolrCore@6fcada url=http://127.0.0.1:49191/unloadcollection1 node=127.0.0.1:49191_ C1848_STATE=coll:unloadcollection core:unloadcollection1 props:{state=down, base_url=http://127.0.0.1:49191, core=unloadcollection1, node_name=127.0.0.1:49191_}
   [junit4]   2> 641562 T3711 C1848 P49191 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49191/unloadcollection1/
   [junit4]   2> 641563 T3711 C1848 P49191 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 641563 T3711 C1848 P49191 oasc.SyncStrategy.syncToMe http://127.0.0.1:49191/unloadcollection1/ has no replicas
   [junit4]   2> 641563 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49191/unloadcollection1/ shard1
   [junit4]   2> 641563 T3711 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leaders/shard1
   [junit4]   2> 641564 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641579 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641582 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641684 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641684 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641684 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641684 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641684 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641684 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641732 T3711 oasc.ZkController.register We are http://127.0.0.1:49191/unloadcollection1/ and leader is http://127.0.0.1:49191/unloadcollection1/
   [junit4]   2> 641733 T3711 oasc.ZkController.register No LogReplay needed for core=unloadcollection1 baseURL=http://127.0.0.1:49191
   [junit4]   2> 641733 T3711 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 641734 T3711 oasc.ZkController.publish publishing core=unloadcollection1 state=active
   [junit4]   2> 641736 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641736 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641736 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641737 T3711 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 641739 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 641739 T3711 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={name=unloadcollection1&version=2&collection=unloadcollection&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891558203unloadcollection1_1n&action=CREATE&numShards=1&wt=javabin} status=0 QTime=1662 
   [junit4]   2> 641740 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49191",
   [junit4]   2> 	  "core":"unloadcollection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49191_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 641744 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641745 T3671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 641747 T3773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d438d name:ZooKeeperConnection Watcher:127.0.0.1:51855/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 641747 T3671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 641749 T3671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 641752 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 641755 T3725 oasc.ZkController.publish publishing core=unloadcollection2 state=down
   [junit4]   2> 641755 T3725 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 641756 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641757 T3725 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 641758 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38768",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38768_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 641758 T3698 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 641758 T3698 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 641760 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 641863 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 641863 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 642757 T3725 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection2
   [junit4]   2> 642758 T3725 oasc.CoreContainer.create Creating SolrCore 'unloadcollection2' using instanceDir: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/unloadcollection2
   [junit4]   2> 642759 T3725 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 642761 T3725 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 642761 T3725 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 642765 T3725 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/unloadcollection2/'
   [junit4]   2> 642795 T3725 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 642834 T3725 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 642837 T3725 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 642847 T3725 oass.IndexSchema.readSchema [unloadcollection2] Schema name=test
   [junit4]   2> 643153 T3725 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 643159 T3725 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 643161 T3725 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 643180 T3725 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 643185 T3725 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 643192 T3725 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 643194 T3725 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 643196 T3725 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 643197 T3725 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 643204 T3725 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 643205 T3725 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 643205 T3725 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 643206 T3725 oasc.SolrCore.<init> [unloadcollection2] Opening new SolrCore at ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388891554470/unloadcollection2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n/
   [junit4]   2> 643206 T3725 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17b24c
   [junit4]   2> 643207 T3725 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n
   [junit4]   2> 643208 T3725 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n/index/
   [junit4]   2> 643208 T3725 oasc.SolrCore.initIndex WARN [unloadcollection2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n/index' doesn't exist. Creating new index...
   [junit4]   2> 643209 T3725 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n/index
   [junit4]   2> 643209 T3725 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=37.3896484375, floorSegmentMB=1.4375, forceMergeDeletesPctAllowed=13.593839988159031, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 643210 T3725 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@da389c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1215a90),segFN=segments_1,generation=1}
   [junit4]   2> 643211 T3725 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 643215 T3725 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 643215 T3725 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 643216 T3725 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 643216 T3725 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 643217 T3725 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 643217 T3725 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 643218 T3725 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 643218 T3725 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 643218 T3725 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 643219 T3725 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 643220 T3725 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 643220 T3725 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 643220 T3725 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 643221 T3725 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 643222 T3725 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 643222 T3725 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 643223 T3725 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 643225 T3725 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 643227 T3725 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 643228 T3725 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 643228 T3725 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=85.58203125, floorSegmentMB=0.53515625, forceMergeDeletesPctAllowed=20.43970905844027, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.17476800285564592
   [junit4]   2> 643229 T3725 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@da389c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1215a90),segFN=segments_1,generation=1}
   [junit4]   2> 643230 T3725 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 643230 T3725 oass.SolrIndexSearcher.<init> Opening Searcher@a7ae94 main
   [junit4]   2> 643233 T3774 oasc.SolrCore.registerSearcher [unloadcollection2] Registered new searcher Searcher@a7ae94 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 643235 T3725 oasc.CoreContainer.registerCore registering core: unloadcollection2
   [junit4]   2> 643235 T3725 oasc.ZkController.register Register replica - core:unloadcollection2 address:http://127.0.0.1:38768 collection:unloadcollection shard:shard1
   [junit4]   2> 643239 T3725 oasc.ZkController.register We are http://127.0.0.1:38768/unloadcollection2/ and leader is http://127.0.0.1:49191/unloadcollection1/
   [junit4]   2> 643240 T3725 oasc.ZkController.register No LogReplay needed for core=unloadcollection2 baseURL=http://127.0.0.1:38768
   [junit4]   2> 643241 T3725 oasc.ZkController.checkRecovery Core needs to recover:unloadcollection2
   [junit4]   2> 643241 T3725 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 643242 T3725 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1849 name=unloadcollection2 org.apache.solr.core.SolrCore@114477b url=http://127.0.0.1:38768/unloadcollection2 node=127.0.0.1:38768_ C1849_STATE=coll:unloadcollection core:unloadcollection2 props:{state=down, base_url=http://127.0.0.1:38768, core=unloadcollection2, node_name=127.0.0.1:38768_}
   [junit4]   2> 643243 T3776 C1849 P38768 oasc.RecoveryStrategy.run Starting recovery process.  core=unloadcollection2 recoveringAfterStartup=true
   [junit4]   2> 643243 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 643244 T3725 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={name=unloadcollection2&version=2&collection=unloadcollection&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-UnloadDistributedZkTest-1388891552490/1388891559879unloadcollection1_2n&action=CREATE&wt=javabin} status=0 QTime=1490 
   [junit4]   2> 643244 T3776 C1849 P38768 oasc.ZkController.publish publishing core=unloadcollection2 state=recovering
   [junit4]   2> 643244 T3776 C1849 P38768 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 643245 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 643246 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 643246 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 643246 T3671 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: unloadcollection failOnTimeout:true timeout (sec):330
   [junit4]   2> 643246 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 643247 T3710 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 643247 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 643247 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 643248 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38768",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38768_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 643255 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 643358 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 643358 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 644248 T3710 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 644248 T3710 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={version=2&checkLive=true&nodeName=127.0.0.1:38768_&onlyIfLeaderActive=true&onlyIfLeader=true&state=recovering&action=PREPRECOVERY&coreNodeName=core_node2&wt=javabin&core=unloadcollection1} status=0 QTime=1001 
   [junit4]   2> 644250 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 645251 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2>  C1849_STATE=coll:unloadcollection core:unloadcollection2 props:{state=recovering, base_url=http://127.0.0.1:38768, core=unloadcollection2, node_name=127.0.0.1:38768_}
   [junit4]   2> 646249 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49191/unloadcollection1/ core=unloadcollection2 - recoveringAfterStartup=true
   [junit4]   2> 646249 T3776 C1849 P38768 oasu.PeerSync.sync PeerSync: core=unloadcollection2 url=http://127.0.0.1:38768 START replicas=[http://127.0.0.1:49191/unloadcollection1/] nUpdates=100
   [junit4]   2> 646250 T3776 C1849 P38768 oasu.PeerSync.sync WARN no frame of reference to tell if we've missed updates
   [junit4]   2> 646251 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=unloadcollection2
   [junit4]   2> 646251 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=unloadcollection2
   [junit4]   2> 646251 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=unloadcollection2
   [junit4]   2> 646251 T3776 C1849 P38768 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 646252 T3776 C1849 P38768 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49191/unloadcollection1/. core=unloadcollection2
   [junit4]   2> 646255 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1850 name=unloadcollection1 org.apache.solr.core.SolrCore@6fcada url=http://127.0.0.1:49191/unloadcollection1 node=127.0.0.1:49191_ C1850_STATE=coll:unloadcollection core:unloadcollection1 props:{state=active, base_url=http://127.0.0.1:49191, core=unloadcollection1, node_name=127.0.0.1:49191_, leader=true}
   [junit4]   2> 646257 T3710 C1850 P49191 oasc.SolrCore.execute [unloadcollection1] webapp= path=/get params={version=2&qt=/get&getVersions=100&distrib=false&wt=javabin} status=0 QTime=1 
   [junit4]   2> 646257 T3709 C1850 P49191 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 646258 T3709 C1850 P49191 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 646258 T3709 C1850 P49191 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 646258 T3709 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&commit=true&waitSearcher=true&commit_end_point=true&softCommit=false&openSearcher=false&wt=javabin} {commit=} 0 1
   [junit4]   2> 646259 T3776 C1849 P38768 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 646259 T3776 C1849 P38768 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 646261 T3712 C1850 P49191 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 646261 T3712 C1850 P49191 oasc.SolrCore.execute [unloadcollection1] webapp= path=/replication params={version=2&qt=/replication&command=indexversion&wt=javabin} status=0 QTime=1 
   [junit4]   2> 646261 T3776 C1849 P38768 oasc.RecoveryStrategy.replay No replay needed. core=unloadcollection2
   [junit4]   2> 646262 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=unloadcollection2
   [junit4]   2> 646262 T3776 C1849 P38768 oasc.ZkController.publish publishing core=unloadcollection2 state=active
   [junit4]   2> 646262 T3776 C1849 P38768 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 646266 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646266 T3776 C1849 P38768 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=unloadcollection2
   [junit4]   2> 646266 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646267 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646268 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 646269 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38768",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38768_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 646271 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 646272 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646272 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646272 T3773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646272 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646272 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646272 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 646273 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647258 T3671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 647259 T3671 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: unloadcollection
   [junit4]   2>  C1849_STATE=coll:unloadcollection core:unloadcollection2 props:{state=active, base_url=http://127.0.0.1:38768, core=unloadcollection2, node_name=127.0.0.1:38768_}
   [junit4]   2> 647269 T3727 C1849 P38768 oasup.LogUpdateProcessor.finish [unloadcollection2] webapp= path=/update params={version=2&distrib.from=http://127.0.0.1:49191/unloadcollection1/&update.distrib=FROMLEADER&wt=javabin} {add=[6 (1456358362067238912)]} 0 2
   [junit4]   2> 647270 T3712 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&wt=javabin} {add=[6 (1456358362067238912)]} 0 9
   [junit4]   2> 647275 T3728 C1849 P38768 oasup.LogUpdateProcessor.finish [unloadcollection2] webapp= path=/update params={version=2&distrib.from=http://127.0.0.1:49191/unloadcollection1/&update.distrib=FROMLEADER&wt=javabin} {add=[7 (1456358362078773248)]} 0 1
   [junit4]   2> 647276 T3710 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&wt=javabin} {add=[7 (1456358362078773248)]} 0 4
   [junit4]   2> 647280 T3725 C1849 P38768 oasup.LogUpdateProcessor.finish [unloadcollection2] webapp= path=/update params={version=2&distrib.from=http://127.0.0.1:49191/unloadcollection1/&update.distrib=FROMLEADER&wt=javabin} {add=[8 (1456358362085064704)]} 0 0
   [junit4]   2> 647281 T3709 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&wt=javabin} {add=[8 (1456358362085064704)]} 0 4
   [junit4]   2> 647284 T3710 C1850 P49191 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 647287 T3710 C1850 P49191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7ad875 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@141d93f),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7ad875 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@141d93f),segFN=segments_2,generation=2}
   [junit4]   2> 647288 T3710 C1850 P49191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 647289 T3710 C1850 P49191 oass.SolrIndexSearcher.<init> Opening Searcher@1ffb19a main
   [junit4]   2> 647290 T3710 C1850 P49191 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 647290 T3769 oasc.SolrCore.registerSearcher [unloadcollection1] Registered new searcher Searcher@1ffb19a main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c3)}
   [junit4]   2> 647291 T3710 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&commit=true&distrib.from=http://127.0.0.1:49191/unloadcollection1/&waitSearcher=true&expungeDeletes=false&commit_end_point=true&softCommit=false&update.distrib=FROMLEADER&wt=javabin} {commit=} 0 7
   [junit4]   2> 647293 T3726 C1849 P38768 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 647295 T3726 C1849 P38768 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@da389c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1215a90),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@da389c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1215a90),segFN=segments_2,generation=2}
   [junit4]   2> 647296 T3726 C1849 P38768 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 647297 T3726 C1849 P38768 oass.SolrIndexSearcher.<init> Opening Searcher@f098b9 main
   [junit4]   2> 647297 T3726 C1849 P38768 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 647298 T3774 oasc.SolrCore.registerSearcher [unloadcollection2] Registered new searcher Searcher@f098b9 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c3)}
   [junit4]   2> 647299 T3726 C1849 P38768 oasup.LogUpdateProcessor.finish [unloadcollection2] webapp= path=/update params={version=2&commit=true&distrib.from=http://127.0.0.1:49191/unloadcollection1/&waitSearcher=true&expungeDeletes=false&commit_end_point=true&softCommit=false&update.distrib=FROMLEADER&wt=javabin} {commit=} 0 6
   [junit4]   2> 647299 T3711 C1850 P49191 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp= path=/update params={version=2&commit=true&waitSearcher=true&softCommit=false&wt=javabin} {commit=} 0 16
   [junit4]   2> 647301 T3742 oasc.ZkController.publish publishing core=unloadcollection3 state=down
   [junit4]   2> 647302 T3742 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 647303 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 647303 T3742 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 647304 T3698 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 647305 T3698 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60336",
   [junit4]   2> 	  "core":"unloadcollection3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60336_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 647305 T3698 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 647305 T3698 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 647307 T3697 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 647409 T3718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647410 T3750 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647410 T3697 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647410 T3773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647409 T3703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647409 T3767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 647409 T3734 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 648303 T3742 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection3
   [junit4]   2> 648304 T3742 oasc.CoreContainer.create Creating SolrCore 'unloadcollection3' using instanceDir: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/unloadcollection3
   [junit4]   2> 648305 T3742 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 648305 T3742 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 648305 T3742 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 648306 T3742 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388891555774/unloadcollection3/'
   [junit4]   2> 648331 T3742 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 648351 T3742 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 648352 T3742 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 648358 T3742 oass.IndexSchema.readSchema [unloadcollection3] Schema name=test
   [junit4]   2> 648574 T3742 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 648578 T3742 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 648580 T3742 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 648590 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 648593 T3742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 648595 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 648596 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 648597 T3742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 648597 T3742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange

[...truncated too long message...]

.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 711427 T3767 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/unloadcollection/leader_elect/shard1/election/91022396784115724-core_node4-n_0000000003
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:276)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 711427 T3767 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 711428 T3767 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78005 T3670 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 98083 T3670 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 98083 T3670 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 101085 T3670 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ar_KW, timezone=Africa/Asmera
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=2,free=149049040,total=429916160
   [junit4]   2> NOTE: All tests run in this JVM: [TestPhraseSuggestions, FieldFacetTest, TestFuzzyAnalyzedSuggestions, TestSchemaNameResource, TestSchemaResource, TestIndexingPerformance, OpenCloseCoreStressTest, ConvertedLegacyTest, TestOmitPositions, ConnectionManagerTest, TestNRTOpen, AliasIntegrationTest, BlockDirectoryTest, TestSolr4Spatial, TestMultiCoreConfBootstrap, TestFileDictionaryLookup, TestDFRSimilarityFactory, IndexSchemaRuntimeFieldTest, TestFieldTypeCollectionResource, TestQuerySenderNoQuery, OpenExchangeRatesOrgProviderTest, HdfsCollectionsAPIDistributedZkTest, DocumentAnalysisRequestHandlerTest, BasicZkTest, RequestHandlersTest, FileUtilsTest, TestHashPartitioner, DocValuesMultiTest, SolrIndexConfigTest, TestZkChroot, SolrPluginUtilsTest, DirectSolrSpellCheckerTest, DefaultValueUpdateProcessorTest, CoreContainerCoreInitFailuresTest, ShardRoutingCustomTest, SolrTestCaseJ4Test, HardAutoCommitTest, ZkSolrClientTest, FieldAnalysisRequestHandlerTest, TestCopyFieldCollectionResource, TestStressReorder, BlockCacheTest, SimplePostToolTest, TestReload, TestSolrDeletionPolicy2, TestBinaryResponseWriter, XmlUpdateRequestHandlerTest, TestCodecSupport, TestFiltering, TestDefaultSearchFieldResource, BadCopyFieldTest, DisMaxRequestHandlerTest, AlternateDirectoryTest, TestSweetSpotSimilarityFactory, TestBinaryField, MultiTermTest, TestStressRecovery, TestMaxScoreQueryParser, TestSolrXmlPersistence, FieldFacetExtrasTest, StressHdfsTest, SpellCheckCollatorTest, TestSolrJ, HighlighterConfigTest, NumericFieldsTest, TestDistributedGrouping, StatsComponentTest, QueryFacetTest, PingRequestHandlerTest, TestFastWriter, TestDistributedSearch, DebugComponentTest, TestIndexSearcher, FullSolrCloudDistribCmdsTest, TestFoldingMultitermQuery, MinimalSchemaTest, DirectSolrConnectionTest, TestClassNameShortening, TestCollationField, TestQueryTypes, TestWordDelimiterFilterFactory, SolrCmdDistributorTest, SolrRequestParserTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=UnloadDistributedZkTest -Dtests.seed=DAF1886D0F823A54 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Africa/Asmera -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | UnloadDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   [junit4]    >    1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DAF1886D0F823A54]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3914, name=Timer-257, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DAF1886D0F823A54]:0)
   [junit4] Completed on J1 in 101.12s, 1 test, 2 errors <<< FAILURES!

[...truncated 532 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 358 suites, 1577 tests, 2 suite-level errors, 678 ignored (6 assumptions)

Total time: 34 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b02 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b02) - Build # 8966 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8966/
Java: 32bit/jdk1.7.0_60-ea-b02 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([E84D964D09D3A5D6:69AB18557E8CC5EA]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.junit.Assert.assertFalse(Assert.java:79)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deleteCollectionWithDownNodes(CollectionsAPIDistributedZkTest.java:390)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:203)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:849)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10181 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 389337 T2323 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 389340 T2323 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1388908099717
   [junit4]   2> 389341 T2323 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 389341 T2324 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 389441 T2323 oasc.ZkTestServer.run start zk server on port:37777
   [junit4]   2> 389442 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389514 T2331 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f25d8e name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389514 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389514 T2323 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 389519 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389522 T2334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b23be name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389523 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389523 T2323 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 389525 T2323 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 389527 T2323 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 389529 T2323 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 389531 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 389531 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 389537 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 389538 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 389540 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 389541 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 389542 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 389543 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 389545 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 389545 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 389547 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 389548 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 389549 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 389550 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 389552 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 389552 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 389554 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 389554 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 389556 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 389557 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 389559 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 389559 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 389566 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389569 T2337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cfaeed name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389570 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389570 T2323 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 389573 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389575 T2340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131e9e1 name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389575 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389576 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 389576 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 389579 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 389579 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 389581 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 389582 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 389584 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 389584 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 389589 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 389590 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 389593 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 389593 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 389595 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 389595 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 389597 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 389598 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 389600 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 389600 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 389602 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 389602 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 389604 T2323 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 389605 T2323 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 389670 T2323 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 389672 T2323 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:३६९७७
   [junit4]   2> 389673 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 389673 T2323 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 389673 T2323 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983
   [junit4]   2> 389673 T2323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/'
   [junit4]   2> 389683 T2323 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solr.xml
   [junit4]   2> 389705 T2323 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 389706 T2323 oasc.CoreContainer.<init> New CoreContainer 26768939
   [junit4]   2> 389707 T2323 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/]
   [junit4]   2> 389708 T2323 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 389708 T2323 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 389708 T2323 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 389709 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 389709 T2323 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 389709 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 389709 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 389710 T2323 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 389710 T2323 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 389712 T2323 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 389712 T2323 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 389712 T2323 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 389713 T2323 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37777/solr
   [junit4]   2> 389713 T2323 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 389714 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389716 T2352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e8ca39 name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389716 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389719 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389720 T2355 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de3e0c name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389720 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389721 T2323 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 389724 T2323 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 389726 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 389730 T2323 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36977_
   [junit4]   2> 389731 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36977_
   [junit4]   2> 389733 T2323 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 389735 T2323 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 389738 T2323 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 389739 T2323 oasc.Overseer.start Overseer (id=91023481223184389-127.0.0.1:36977_-n_0000000000) starting
   [junit4]   2> 389742 T2323 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 389744 T2357 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 389744 T2323 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 389747 T2323 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 389749 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 389751 T2356 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 389752 T2323 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983
   [junit4]   2> 389753 T2323 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/conf/
   [junit4]   2> 389790 T2323 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 389791 T2323 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 389791 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 389792 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389794 T2361 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e096cf name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389795 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389796 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 389798 T2323 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 389859 T2323 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 389861 T2323 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:५८५६१
   [junit4]   2> 389861 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 389862 T2323 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 389862 T2323 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174
   [junit4]   2> 389862 T2323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/'
   [junit4]   2> 389872 T2323 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solr.xml
   [junit4]   2> 389900 T2323 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 389900 T2323 oasc.CoreContainer.<init> New CoreContainer 22067291
   [junit4]   2> 389900 T2323 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/]
   [junit4]   2> 389901 T2323 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 389901 T2323 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 389902 T2323 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 389902 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 389902 T2323 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 389902 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 389903 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 389903 T2323 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 389903 T2323 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 389905 T2323 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 389905 T2323 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 389906 T2323 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 389906 T2323 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37777/solr
   [junit4]   2> 389906 T2323 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 389907 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389908 T2373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bfa3e name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389909 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389911 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 389912 T2376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e5a20 name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 389912 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 389915 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 390917 T2323 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58561_
   [junit4]   2> 390918 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58561_
   [junit4]   2> 390921 T2361 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 390921 T2355 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 390921 T2376 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 390925 T2323 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174
   [junit4]   2> 390925 T2323 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/conf/
   [junit4]   2> 390950 T2323 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 390950 T2323 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 390950 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 391010 T2323 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 391011 T2323 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:५६७२०
   [junit4]   2> 391012 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 391012 T2323 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 391012 T2323 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327
   [junit4]   2> 391013 T2323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327/'
   [junit4]   2> 391022 T2323 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327/solr.xml
   [junit4]   2> 391041 T2323 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 391042 T2323 oasc.CoreContainer.<init> New CoreContainer 4918717
   [junit4]   2> 391042 T2323 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327/]
   [junit4]   2> 391043 T2323 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 391043 T2323 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 391043 T2323 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 391043 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 391044 T2323 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 391044 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 391044 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 391044 T2323 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 391045 T2323 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 391047 T2323 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 391047 T2323 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 391047 T2323 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 391047 T2323 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37777/solr
   [junit4]   2> 391048 T2323 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 391049 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 391050 T2389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e767a0 name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 391050 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 391053 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 391054 T2392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14f0398 name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 391055 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 391058 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 392060 T2323 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56720_
   [junit4]   2> 392061 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56720_
   [junit4]   2> 392064 T2376 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 392064 T2361 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 392064 T2355 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 392065 T2392 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 392068 T2323 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327
   [junit4]   2> 392069 T2323 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1388908101327/conf/
   [junit4]   2> 392089 T2323 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 392090 T2323 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 392090 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 392151 T2323 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 392153 T2323 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:३३७८७
   [junit4]   2> 392153 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 392154 T2323 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 392154 T2323 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467
   [junit4]   2> 392154 T2323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/'
   [junit4]   2> 392164 T2323 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solr.xml
   [junit4]   2> 392184 T2323 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 392184 T2323 oasc.CoreContainer.<init> New CoreContainer 32488535
   [junit4]   2> 392185 T2323 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/]
   [junit4]   2> 392185 T2323 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 392185 T2323 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 392186 T2323 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 392186 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 392186 T2323 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 392186 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 392187 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 392187 T2323 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 392187 T2323 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 392189 T2323 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 392189 T2323 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 392190 T2323 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 392190 T2323 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37777/solr
   [junit4]   2> 392190 T2323 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 392191 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 392192 T2405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cfba4c name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 392193 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 392194 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 392195 T2408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10510fa name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 392195 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 392198 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 393201 T2323 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33787_
   [junit4]   2> 393203 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33787_
   [junit4]   2> 393209 T2376 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 393209 T2408 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 393209 T2361 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 393209 T2392 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 393209 T2355 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 393217 T2323 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467
   [junit4]   2> 393219 T2323 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/conf/
   [junit4]   2> 393251 T2323 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 393252 T2323 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 393252 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 393313 T2323 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 393316 T2323 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:३९७६०
   [junit4]   2> 393316 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 393316 T2323 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 393317 T2323 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629
   [junit4]   2> 393317 T2323 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/'
   [junit4]   2> 393334 T2323 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solr.xml
   [junit4]   2> 393358 T2323 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 393358 T2323 oasc.CoreContainer.<init> New CoreContainer 7220004
   [junit4]   2> 393358 T2323 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/]
   [junit4]   2> 393359 T2323 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 393359 T2323 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 393360 T2323 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 393360 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 393360 T2323 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 393360 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 393361 T2323 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 393361 T2323 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 393361 T2323 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 393363 T2323 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 393363 T2323 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 393363 T2323 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 393364 T2323 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37777/solr
   [junit4]   2> 393364 T2323 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 393365 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393367 T2421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e56815 name:ZooKeeperConnection Watcher:127.0.0.1:37777 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393368 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393372 T2323 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 393373 T2424 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1605817 name:ZooKeeperConnection Watcher:127.0.0.1:37777/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 393374 T2323 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 393379 T2323 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 394384 T2323 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39760_
   [junit4]   2> 394387 T2323 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39760_
   [junit4]   2> 394392 T2408 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394393 T2361 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394393 T2355 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394394 T2392 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394393 T2376 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394393 T2424 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 394400 T2323 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629
   [junit4]   2> 394402 T2323 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/conf/
   [junit4]   2> 394430 T2323 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 394431 T2323 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 394431 T2323 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 394440 T2370 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&collection.configName=conf1&action=CREATE&router.field=myOwnField&name=solrj_collection&router.name=compositeId&numShards=2&replicationFactor=2&wt=javabin
   [junit4]   2> 394441 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394442 T2357 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 394442 T2357 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 394442 T2357 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 394443 T2357 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 394445 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394446 T2356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 394447 T2356 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 394447 T2356 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 394449 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394449 T2408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394449 T2361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394449 T2355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394449 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394449 T2392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394449 T2424 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394545 T2357 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 394546 T2357 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:33787_
   [junit4]   2> 394546 T2357 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:58561_
   [junit4]   2> 394547 T2357 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:39760_
   [junit4]   2> 394547 T2357 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:36977_
   [junit4]   2> 394549 T2367 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down
   [junit4]   2> 394549 T2400 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down
   [junit4]   2> 394549 T2347 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down
   [junit4]   2> 394549 T2417 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down
   [junit4]   2> 394550 T2367 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 394550 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394551 T2400 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 394551 T2417 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 394551 T2347 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 394552 T2356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 394553 T2356 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58561",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58561_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 394555 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394555 T2356 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33787",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33787_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 394557 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394558 T2356 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36977",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36977_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 394560 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394560 T2356 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39760",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39760_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 394562 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 394665 T2408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394665 T2361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394665 T2355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394665 T2424 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394665 T2392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 394665 T2376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 395551 T2367 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 395552 T2400 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 395552 T2347 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 395552 T2367 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2
   [junit4]   2> 395553 T2367 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 395552 T2417 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 395553 T2347 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2
   [junit4]   2> 395553 T2400 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1
   [junit4]   2> 395555 T2400 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 395555 T2417 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1
   [junit4]   2> 395555 T2347 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 395555 T2367 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 395557 T2367 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 395556 T2400 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 395558 T2400 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 395556 T2417 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 395559 T2367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/'
   [junit4]   2> 395557 T2347 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 395560 T2400 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/'
   [junit4]   2> 395560 T2417 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 395560 T2347 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 395561 T2417 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 395563 T2347 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/'
   [junit4]   2> 395565 T2417 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/'
   [junit4]   2> 395607 T2417 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 395607 T2367 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 395609 T2347 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 395622 T2400 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 395638 T2417 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 395639 T2417 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 395640 T2367 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 395641 T2367 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 395645 T2417 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 395646 T2347 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 395646 T2367 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 395647 T2347 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 395656 T2400 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 395657 T2347 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 395658 T2400 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 395668 T2400 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 395870 T2417 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 395875 T2417 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 395882 T2417 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 395891 T2367 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 395893 T2417 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395895 T2367 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 395895 T2417 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395897 T2367 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 395898 T2417 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395899 T2417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395901 T2417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395901 T2417 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395902 T2417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395903 T2417 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395903 T2417 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 395903 T2417 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/data/
   [junit4]   2> 395903 T2417 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f8f9b9
   [junit4]   2> 395904 T2417 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/data
   [junit4]   2> 395905 T2417 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 395905 T2417 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 395905 T2417 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1388908103629/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 395906 T2417 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=42.310546875, floorSegmentMB=1.2890625, forceMergeDeletesPctAllowed=3.6441658803137997, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8820327101017023
   [junit4]   2> 395906 T2417 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10261b9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10b344d),segFN=segments_1,generation=1}
   [junit4]   2> 395907 T2417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395908 T2367 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395909 T2417 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 395910 T2417 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 395910 T2417 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 395910 T2367 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395911 T2417 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395911 T2417 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395911 T2417 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 395912 T2417 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395912 T2417 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395912 T2417 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 395913 T2417 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 395913 T2367 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395914 T2417 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 395914 T2417 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 395914 T2367 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395915 T2417 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 395915 T2367 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395915 T2417 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 395915 T2367 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395916 T2417 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 395916 T2417 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 395917 T2417 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 395917 T2367 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395917 T2367 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395918 T2367 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 395918 T2367 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/data/
   [junit4]   2> 395918 T2367 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f8f9b9
   [junit4]   2> 395919 T2367 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/data
   [junit4]   2> 395919 T2417 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 395919 T2367 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 395920 T2367 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 395920 T2367 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1388908100174/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 395920 T2367 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=42.310546875, floorSegmentMB=1.2890625, forceMergeDeletesPctAllowed=3.6441658803137997, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8820327101017023
   [junit4]   2> 395921 T2367 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@828ce4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1da959d),segFN=segments_1,generation=1}
   [junit4]   2> 395921 T2417 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 395921 T2367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395922 T2417 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 395922 T2417 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23339759227448056]
   [junit4]   2> 395923 T2417 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@10261b9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10b344d),segFN=segments_1,generation=1}
   [junit4]   2> 395923 T2417 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395923 T2417 oass.SolrIndexSearcher.<init> Opening Searcher@9b5d63 main
   [junit4]   2> 395923 T2347 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 395924 T2367 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 395925 T2367 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 395925 T2367 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 395925 T2367 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395925 T2430 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@9b5d63 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 395926 T2367 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395926 T2367 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 395926 T2367 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395927 T2417 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica1
   [junit4]   2> 395927 T2367 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395927 T2417 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica1 address:http://127.0.0.1:39760 collection:solrj_collection shard:shard2
   [junit4]   2> 395927 T2367 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 395928 T2367 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 395928 T2367 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 395928 T2417 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 395928 T2367 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 395929 T2367 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 395929 T2347 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 395929 T2367 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 395930 T2367 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 395930 T2367 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 395931 T2367 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 395932 T2367 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 395932 T2347 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 395933 T2417 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 395933 T2367 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 395934 T2367 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 395934 T2367 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23339759227448056]
   [junit4]   2> 395935 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395935 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395935 T2367 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@828ce4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1da959d),segFN=segments_1,generation=1}
   [junit4]   2> 395935 T2367 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395935 T2417 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=180000
   [junit4]   2> 395935 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395936 T2356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 395935 T2367 oass.SolrIndexSearcher.<init> Opening Searcher@17b2667 main
   [junit4]   2> 395938 T2432 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@17b2667 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 395938 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395940 T2367 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 395940 T2367 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:http://127.0.0.1:58561 collection:solrj_collection shard:shard1
   [junit4]   2> 395941 T2367 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 395943 T2347 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395945 T2367 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 395946 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395947 T2367 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 395947 T2347 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 395949 T2355 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 395951 T2347 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395952 T2347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395952 T2347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395952 T2347 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 395954 T2347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 395954 T2347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 395954 T2347 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 395955 T2347 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/data/
   [junit4]   2> 395955 T2347 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f8f9b9
   [junit4]   2> 395956 T2347 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/data
   [junit4]   2> 395957 T2347 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 395957 T2347 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 395957 T2347 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1388908099983/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 395958 T2347 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=42.310546875, floorSegmentMB=1.2890625, forceMergeDeletesPctAllowed=3.6441658803137997, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8820327101017023
   [junit4]   2> 395959 T2347 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1277f77 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@162a6f),segFN=segments_1,generation=1}
   [junit4]   2> 395959 T2347 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395963 T2347 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 395964 T2347 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 395964 T2347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 395964 T2347 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395965 T2347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 395965 T2347 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 395966 T2347 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395966 T2347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 395966 T2347 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 395967 T2347 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 395968 T2347 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 395968 T2347 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 395968 T2347 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 395969 T2347 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 395969 T2347 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 395970 T2347 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 395970 T2347 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 395972 T2347 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 395975 T2347 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 395975 T2347 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 395976 T2347 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.23339759227448056]
   [junit4]   2> 395976 T2347 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1277f77 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@162a6f),segFN=segments_1,generation=1}
   [junit4]   2> 395977 T2347 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 395977 T2347 oass.SolrIndexSearcher.<init> Opening Searcher@b14ecb main
   [junit4]   2> 395980 T2434 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@b14ecb main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 395982 T2347 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 395983 T2347 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:36977 collection:solrj_collection shard:shard2
   [junit4]   2> 395986 T2400 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 395990 T2400 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 395992 T2400 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 396001 T2400 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 396004 T2400 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 396007 T2400 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 396007 T2400 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 396008 T2400 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 396008 T2400 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 396009 T2400 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 396009 T2400 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 396009 T2400 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 396010 T2400 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/, dataDir=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/data/
   [junit4]   2> 396010 T2400 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f8f9b9
   [junit4]   2> 396011 T2400 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/data
   [junit4]   2> 396011 T2400 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 396011 T2400 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 396011 T2400 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 396012 T2400 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=

[...truncated too long message...]

ng main searcher on request.
   [junit4]   2> 496287 T2408 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 496287 T2408 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data/index;done=false>>]
   [junit4]   2> 496288 T2408 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data/index
   [junit4]   2> 496288 T2408 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data;done=false>>]
   [junit4]   2> 496288 T2408 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1388908102467/awholynewcollection_1_shard1_replica1/data
   [junit4]   2> 496288 T2408 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1/leaders/shard1
   [junit4]   2> 496289 T2408 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 496289 T2408 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_1/leader_elect/shard1/election/91023481223184396-core_node1-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:276)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 496290 T2408 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 496290 T2408 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=hi_IN, timezone=Asia/Baghdad
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=186587856,total=328466432
   [junit4]   2> NOTE: All tests run in this JVM: [RequiredFieldsTest, SignatureUpdateProcessorFactoryTest, TestFastWriter, HighlighterConfigTest, SimplePostToolTest, TestFieldTypeResource, CoreContainerCoreInitFailuresTest, TestCSVLoader, TestSweetSpotSimilarityFactory, TestRecovery, PreAnalyzedUpdateProcessorTest, TestDistributedSearch, XmlUpdateRequestHandlerTest, DebugComponentTest, DocValuesMultiTest, CurrencyFieldOpenExchangeTest, DistributedSuggestComponentTest, TestClassNameShortening, TestDFRSimilarityFactory, StressHdfsTest, SolrTestCaseJ4Test, NumericFieldsTest, FullSolrCloudDistribCmdsTest, TestPhraseSuggestions, TestSerializedLuceneMatchVersion, TestStressUserVersions, SpellCheckComponentTest, SuggesterTSTTest, AbstractAnalyticsStatsTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestIBSimilarityFactory, TestCollationField, TimeZoneUtilsTest, TestUtils, BadCopyFieldTest, ConnectionManagerTest, TestStressLucene, PluginInfoTest, TestSimpleQParserPlugin, IndexSchemaRuntimeFieldTest, TestDistributedGrouping, TestMultiCoreConfBootstrap, TestRandomFaceting, TestHashPartitioner, AssignTest, TestFieldTypeCollectionResource, HardAutoCommitTest, ChaosMonkeySafeLeaderTest, TestStressRecovery, TermVectorComponentDistributedTest, ChaosMonkeyNothingIsSafeTest, PingRequestHandlerTest, SolrPluginUtilsTest, XsltUpdateRequestHandlerTest, TestSolrCoreProperties, TestDistribDocBasedVersion, TestRandomMergePolicy, TestFileDictionaryLookup, NoCacheHeaderTest, SolrRequestParserTest, AlternateDirectoryTest, TestIndexSearcher, MigrateRouteKeyTest, TestSchemaVersionResource, TestSchemaResource, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 107.77s, 1 test, 1 failure <<< FAILURES!

[...truncated 737 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 358 suites, 1577 tests, 1 failure, 35 ignored (6 assumptions)

Total time: 38 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b02 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_45) - Build # 8965 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8965/
Java: 32bit/jdk1.7.0_45 -client -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.UnloadDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest:     1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([162A6C54E66BCAC3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.UnloadDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([162A6C54E66BCAC3]:0)




Build Log:
[...truncated 10389 lines...]
   [junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
   [junit4]   2> 824932 T3989 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nl_eln/
   [junit4]   2> 824935 T3989 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553
   [junit4]   2> 824936 T3989 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 824936 T3990 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 825036 T3989 oasc.ZkTestServer.run start zk server on port:33087
   [junit4]   2> 825037 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825075 T3997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1416524 name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825076 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825076 T3989 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 825079 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825080 T4000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@193ebe4 name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825080 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825081 T3989 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 825083 T3989 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 825085 T3989 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 825086 T3989 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 825088 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 825088 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 825091 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 825091 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 825094 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 825094 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 825096 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 825097 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 825099 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 825099 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 825101 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 825102 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 825104 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 825105 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 825107 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 825107 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 825109 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 825109 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 825111 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 825112 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 825114 T3989 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 825114 T3989 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 825189 T3989 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 825192 T3989 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44697
   [junit4]   2> 825192 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 825192 T3989 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 825193 T3989 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734
   [junit4]   2> 825193 T3989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734/'
   [junit4]   2> 825205 T3989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734/solr.xml
   [junit4]   2> 825246 T3989 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 825246 T3989 oasc.CoreContainer.<init> New CoreContainer 18679530
   [junit4]   2> 825246 T3989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734/]
   [junit4]   2> 825248 T3989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 825248 T3989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 825248 T3989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 825249 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 825249 T3989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 825249 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 825249 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 825250 T3989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 825250 T3989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 825252 T3989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 825253 T3989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 825253 T3989 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 825253 T3989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33087/solr
   [junit4]   2> 825254 T3989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 825254 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825255 T4012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f41f4 name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825256 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825257 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825258 T4015 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@152d3a8 name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825259 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825260 T3989 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 825262 T3989 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 825264 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 825265 T3989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44697_nl_eln
   [junit4]   2> 825266 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44697_nl_eln
   [junit4]   2> 825268 T3989 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 825269 T3989 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 825274 T3989 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 825277 T3989 oasc.Overseer.start Overseer (id=91023158447570947-127.0.0.1:44697_nl_eln-n_0000000000) starting
   [junit4]   2> 825280 T3989 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 825282 T4017 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 825283 T3989 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 825285 T3989 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 825286 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 825288 T4016 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 825290 T3989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734
   [junit4]   2> 825291 T3989 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-controljetty-1388903174734/conf/
   [junit4]   2> 825322 T3989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 825323 T3989 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 825323 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 825324 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825325 T4021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@178bd2c name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825325 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825326 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 825327 T3989 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 825394 T3989 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 825401 T3989 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57041
   [junit4]   2> 825402 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 825402 T3989 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 825403 T3989 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945
   [junit4]   2> 825403 T3989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/'
   [junit4]   2> 825414 T3989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/solr.xml
   [junit4]   2> 825455 T3989 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 825456 T3989 oasc.CoreContainer.<init> New CoreContainer 2561508
   [junit4]   2> 825456 T3989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/]
   [junit4]   2> 825457 T3989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 825458 T3989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 825458 T3989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 825458 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 825458 T3989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 825459 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 825459 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 825459 T3989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 825460 T3989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 825462 T3989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 825462 T3989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 825462 T3989 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 825463 T3989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33087/solr
   [junit4]   2> 825463 T3989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 825464 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825465 T4033 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@509d9b name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825465 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825467 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825468 T4036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15ca86b name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825469 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825471 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 826474 T3989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57041_nl_eln
   [junit4]   2> 826475 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57041_nl_eln
   [junit4]   2> 826477 T4021 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 826478 T4036 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 826478 T4015 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 826483 T3989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945
   [junit4]   2> 826484 T3989 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/conf/
   [junit4]   2> 826515 T3989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 826516 T3989 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 826516 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 826583 T3989 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 826584 T3989 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37740
   [junit4]   2> 826585 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 826585 T3989 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 826586 T3989 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134
   [junit4]   2> 826586 T3989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/'
   [junit4]   2> 826604 T3989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/solr.xml
   [junit4]   2> 826645 T3989 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 826646 T3989 oasc.CoreContainer.<init> New CoreContainer 16903029
   [junit4]   2> 826646 T3989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/]
   [junit4]   2> 826647 T3989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 826648 T3989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 826648 T3989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 826648 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 826649 T3989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 826649 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 826649 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 826650 T3989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 826650 T3989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 826652 T3989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 826652 T3989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 826653 T3989 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 826653 T3989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33087/solr
   [junit4]   2> 826653 T3989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 826654 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 826655 T4049 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31df9e name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 826655 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 826657 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 826658 T4052 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@850235 name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 826658 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 826661 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 827663 T3989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37740_nl_eln
   [junit4]   2> 827665 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37740_nl_eln
   [junit4]   2> 827667 T4036 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 827667 T4015 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 827667 T4021 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 827667 T4052 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 827673 T3989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134
   [junit4]   2> 827675 T3989 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/conf/
   [junit4]   2> 827708 T3989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 827708 T3989 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 827709 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 827780 T3989 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 827782 T3989 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39913
   [junit4]   2> 827782 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 827783 T3989 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 827783 T3989 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326
   [junit4]   2> 827783 T3989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/'
   [junit4]   2> 827795 T3989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/solr.xml
   [junit4]   2> 827842 T3989 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 827843 T3989 oasc.CoreContainer.<init> New CoreContainer 15346592
   [junit4]   2> 827843 T3989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/]
   [junit4]   2> 827844 T3989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 827845 T3989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 827845 T3989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 827845 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 827846 T3989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 827846 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 827846 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 827847 T3989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 827847 T3989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 827849 T3989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 827849 T3989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 827850 T3989 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 827850 T3989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33087/solr
   [junit4]   2> 827850 T3989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 827851 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 827852 T4065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ad06a8 name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 827852 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 827854 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 827855 T4068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14b592 name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 827855 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 827858 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 828860 T3989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39913_nl_eln
   [junit4]   2> 828862 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39913_nl_eln
   [junit4]   2> 828865 T4021 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 828865 T4015 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 828865 T4036 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 828865 T4052 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 828866 T4068 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 828870 T3989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326
   [junit4]   2> 828871 T3989 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/conf/
   [junit4]   2> 828904 T3989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 828905 T3989 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 828905 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 828972 T3989 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 828974 T3989 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35436
   [junit4]   2> 828975 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 828975 T3989 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 828975 T3989 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523
   [junit4]   2> 828976 T3989 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523/'
   [junit4]   2> 828988 T3989 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523/solr.xml
   [junit4]   2> 829036 T3989 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 829036 T3989 oasc.CoreContainer.<init> New CoreContainer 10534509
   [junit4]   2> 829036 T3989 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523/]
   [junit4]   2> 829038 T3989 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 829038 T3989 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 829038 T3989 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 829038 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 829039 T3989 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 829039 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 829039 T3989 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 829040 T3989 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 829040 T3989 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 829042 T3989 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 829042 T3989 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 829043 T3989 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 829043 T3989 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33087/solr
   [junit4]   2> 829043 T3989 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 829044 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 829046 T4081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13099c9 name:ZooKeeperConnection Watcher:127.0.0.1:33087 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 829046 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 829048 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 829049 T4084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e42aa name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 829049 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 829052 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 830054 T3989 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35436_nl_eln
   [junit4]   2> 830055 T3989 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35436_nl_eln
   [junit4]   2> 830058 T4068 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830058 T4021 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830058 T4015 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830059 T4052 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830058 T4036 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830059 T4084 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 830063 T3989 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523
   [junit4]   2> 830065 T3989 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty4-1388903178523/conf/
   [junit4]   2> 830089 T3989 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 830089 T3989 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 830090 T3989 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 830092 T4029 oasc.ZkController.publish publishing core=unloadcollection1 state=down
   [junit4]   2> 830093 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 830093 T4029 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 830094 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 830095 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57041/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57041_nl_eln",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 830095 T4016 oasc.Overseer$ClusterStateUpdater.createCollection Create collection unloadcollection with shards [shard1]
   [junit4]   2> 830095 T4016 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 830097 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 830099 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 830099 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 830099 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 830099 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 830099 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 830099 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831094 T4029 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection1
   [junit4]   2> 831095 T4029 oasc.CoreContainer.create Creating SolrCore 'unloadcollection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/unloadcollection1
   [junit4]   2> 831095 T4029 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 831096 T4029 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:unloadcollection
   [junit4]   2> 831096 T4029 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 831097 T4029 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 831097 T4029 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection
   [junit4]   2> 831099 T4029 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 831100 T4029 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/unloadcollection1/'
   [junit4]   2> 831141 T4029 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 831189 T4029 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 831190 T4029 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 831198 T4029 oass.IndexSchema.readSchema [unloadcollection1] Schema name=test
   [junit4]   2> 831614 T4029 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 831621 T4029 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 831624 T4029 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 831640 T4029 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 831643 T4029 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 831646 T4029 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 831647 T4029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 831647 T4029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 831648 T4029 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 831648 T4029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 831649 T4029 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 831649 T4029 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 831649 T4029 oasc.SolrCore.<init> [unloadcollection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty1-1388903174945/unloadcollection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n/
   [junit4]   2> 831650 T4029 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5b6afd
   [junit4]   2> 831651 T4029 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n
   [junit4]   2> 831651 T4029 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n/index/
   [junit4]   2> 831651 T4029 oasc.SolrCore.initIndex WARN [unloadcollection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n/index' doesn't exist. Creating new index...
   [junit4]   2> 831652 T4029 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n/index
   [junit4]   2> 831652 T4029 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.410373319039952]
   [junit4]   2> 831653 T4029 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@18fd9da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1dadf0d),segFN=segments_1,generation=1}
   [junit4]   2> 831653 T4029 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 831656 T4029 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 831656 T4029 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 831657 T4029 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 831657 T4029 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 831657 T4029 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 831657 T4029 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 831658 T4029 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 831658 T4029 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 831659 T4029 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 831659 T4029 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 831660 T4029 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 831660 T4029 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 831660 T4029 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 831661 T4029 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 831661 T4029 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 831662 T4029 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 831662 T4029 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 831665 T4029 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 831667 T4029 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 831668 T4029 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 831668 T4029 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=11.9677734375, floorSegmentMB=2.181640625, forceMergeDeletesPctAllowed=22.612758632362407, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 831669 T4029 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@18fd9da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1dadf0d),segFN=segments_1,generation=1}
   [junit4]   2> 831669 T4029 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 831669 T4029 oass.SolrIndexSearcher.<init> Opening Searcher@1b13319 main
   [junit4]   2> 831673 T4086 oasc.SolrCore.registerSearcher [unloadcollection1] Registered new searcher Searcher@1b13319 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 831674 T4029 oasc.CoreContainer.registerCore registering core: unloadcollection1
   [junit4]   2> 831675 T4029 oasc.ZkController.register Register replica - core:unloadcollection1 address:http://127.0.0.1:57041/nl_eln collection:unloadcollection shard:shard1
   [junit4]   2> 831676 T4029 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leader_elect/shard1/election
   [junit4]   2> 831680 T4029 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 831683 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831683 T4029 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 831684 T4029 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C6165 name=unloadcollection1 org.apache.solr.core.SolrCore@1500442 url=http://127.0.0.1:57041/nl_eln/unloadcollection1 node=127.0.0.1:57041_nl_eln C6165_STATE=coll:unloadcollection core:unloadcollection1 props:{state=down, base_url=http://127.0.0.1:57041/nl_eln, core=unloadcollection1, node_name=127.0.0.1:57041_nl_eln}
   [junit4]   2> 831684 T4029 C6165 P57041 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57041/nl_eln/unloadcollection1/
   [junit4]   2> 831685 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 831685 T4029 C6165 P57041 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 831685 T4029 C6165 P57041 oasc.SyncStrategy.syncToMe http://127.0.0.1:57041/nl_eln/unloadcollection1/ has no replicas
   [junit4]   2> 831686 T4029 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57041/nl_eln/unloadcollection1/ shard1
   [junit4]   2> 831686 T4029 oascc.SolrZkClient.makePath makePath: /collections/unloadcollection/leaders/shard1
   [junit4]   2> 831688 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831689 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831689 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831689 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831689 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831689 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831689 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831693 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831694 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 831697 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831798 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831799 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831799 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831798 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831798 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831799 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831844 T4029 oasc.ZkController.register We are http://127.0.0.1:57041/nl_eln/unloadcollection1/ and leader is http://127.0.0.1:57041/nl_eln/unloadcollection1/
   [junit4]   2> 831845 T4029 oasc.ZkController.register No LogReplay needed for core=unloadcollection1 baseURL=http://127.0.0.1:57041/nl_eln
   [junit4]   2> 831845 T4029 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 831845 T4029 oasc.ZkController.publish publishing core=unloadcollection1 state=active
   [junit4]   2> 831846 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831847 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831847 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831847 T4029 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 831848 T4029 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={wt=javabin&version=2&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903179707unloadcollection1_1n&numShards=1&collection=unloadcollection&action=CREATE&name=unloadcollection1} status=0 QTime=1757 
   [junit4]   2> 831848 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 831849 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57041/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57041_nl_eln",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 831849 T3989 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 831850 T4090 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b361c2 name:ZooKeeperConnection Watcher:127.0.0.1:33087/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 831851 T3989 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 831851 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831852 T3989 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 831853 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 831855 T4046 oasc.ZkController.publish publishing core=unloadcollection2 state=down
   [junit4]   2> 831855 T4046 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 831856 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831856 T4046 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 831857 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37740/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37740_nl_eln",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 831857 T4016 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 831858 T4016 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 831859 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 831962 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831963 T4090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831963 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831963 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831963 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831963 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 831962 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 832857 T4046 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection2
   [junit4]   2> 832858 T4046 oasc.CoreContainer.create Creating SolrCore 'unloadcollection2' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/unloadcollection2
   [junit4]   2> 832858 T4046 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 832859 T4046 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 832859 T4046 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 832860 T4046 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/unloadcollection2/'
   [junit4]   2> 832899 T4046 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 832950 T4046 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 832952 T4046 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 832960 T4046 oass.IndexSchema.readSchema [unloadcollection2] Schema name=test
   [junit4]   2> 833376 T4046 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 833383 T4046 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 833386 T4046 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 833397 T4046 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 833402 T4046 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 833407 T4046 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 833408 T4046 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 833409 T4046 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 833409 T4046 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 833410 T4046 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 833411 T4046 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 833411 T4046 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 833412 T4046 oasc.SolrCore.<init> [unloadcollection2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty2-1388903176134/unloadcollection2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n/
   [junit4]   2> 833412 T4046 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5b6afd
   [junit4]   2> 833413 T4046 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n
   [junit4]   2> 833414 T4046 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n/index/
   [junit4]   2> 833414 T4046 oasc.SolrCore.initIndex WARN [unloadcollection2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n/index' doesn't exist. Creating new index...
   [junit4]   2> 833415 T4046 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n/index
   [junit4]   2> 833415 T4046 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=48, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.410373319039952]
   [junit4]   2> 833416 T4046 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ddf88 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11f9052),segFN=segments_1,generation=1}
   [junit4]   2> 833417 T4046 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 833421 T4046 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 833422 T4046 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 833423 T4046 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 833423 T4046 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 833424 T4046 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 833424 T4046 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 833425 T4046 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 833425 T4046 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 833426 T4046 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 833427 T4046 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 833428 T4046 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 833428 T4046 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 833428 T4046 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 833429 T4046 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 833430 T4046 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 833430 T4046 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 833431 T4046 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 833433 T4046 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 833445 T4046 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 833446 T4046 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 833447 T4046 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=11.9677734375, floorSegmentMB=2.181640625, forceMergeDeletesPctAllowed=22.612758632362407, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 833448 T4046 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3ddf88 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11f9052),segFN=segments_1,generation=1}
   [junit4]   2> 833448 T4046 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 833449 T4046 oass.SolrIndexSearcher.<init> Opening Searcher@170d8b6 main
   [junit4]   2> 833452 T4091 oasc.SolrCore.registerSearcher [unloadcollection2] Registered new searcher Searcher@170d8b6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 833457 T4046 oasc.CoreContainer.registerCore registering core: unloadcollection2
   [junit4]   2> 833457 T4046 oasc.ZkController.register Register replica - core:unloadcollection2 address:http://127.0.0.1:37740/nl_eln collection:unloadcollection shard:shard1
   [junit4]   2> 833460 T4046 oasc.ZkController.register We are http://127.0.0.1:37740/nl_eln/unloadcollection2/ and leader is http://127.0.0.1:57041/nl_eln/unloadcollection1/
   [junit4]   2> 833460 T4046 oasc.ZkController.register No LogReplay needed for core=unloadcollection2 baseURL=http://127.0.0.1:37740/nl_eln
   [junit4]   2> 833460 T4046 oasc.ZkController.checkRecovery Core needs to recover:unloadcollection2
   [junit4]   2> 833461 T4046 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C6166 name=unloadcollection2 org.apache.solr.core.SolrCore@1eb0846 url=http://127.0.0.1:37740/nl_eln/unloadcollection2 node=127.0.0.1:37740_nl_eln C6166_STATE=coll:unloadcollection core:unloadcollection2 props:{state=down, base_url=http://127.0.0.1:37740/nl_eln, core=unloadcollection2, node_name=127.0.0.1:37740_nl_eln}
   [junit4]   2> 833461 T4093 C6166 P37740 oasc.RecoveryStrategy.run Starting recovery process.  core=unloadcollection2 recoveringAfterStartup=true
   [junit4]   2> 833462 T4046 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 833462 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 833463 T4093 C6166 P37740 oasc.ZkController.publish publishing core=unloadcollection2 state=recovering
   [junit4]   2> 833463 T4093 C6166 P37740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 833463 T4046 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={wt=javabin&version=2&dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-UnloadDistributedZkTest-1388903174553/1388903181471unloadcollection1_2n&collection=unloadcollection&action=CREATE&name=unloadcollection2} status=0 QTime=1608 
   [junit4]   2> 833464 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 833464 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 833464 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 833464 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 833465 T4030 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 833466 T3989 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: unloadcollection failOnTimeout:true timeout (sec):330
   [junit4]   2> 833466 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 833467 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 833467 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37740/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37740_nl_eln",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 833469 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 833470 T4090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 833470 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 834466 T4030 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 834466 T4030 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={wt=javabin&version=2&action=PREPRECOVERY&onlyIfLeaderActive=true&core=unloadcollection1&coreNodeName=core_node2&checkLive=true&onlyIfLeader=true&nodeName=127.0.0.1:37740_nl_eln&state=recovering} status=0 QTime=1001 
   [junit4]   2> 834468 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 835470 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2>  C6166_STATE=coll:unloadcollection core:unloadcollection2 props:{state=recovering, base_url=http://127.0.0.1:37740/nl_eln, core=unloadcollection2, node_name=127.0.0.1:37740_nl_eln}
   [junit4]   2> 836467 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:57041/nl_eln/unloadcollection1/ core=unloadcollection2 - recoveringAfterStartup=true
   [junit4]   2> 836468 T4093 C6166 P37740 oasu.PeerSync.sync PeerSync: core=unloadcollection2 url=http://127.0.0.1:37740/nl_eln START replicas=[http://127.0.0.1:57041/nl_eln/unloadcollection1/] nUpdates=100
   [junit4]   2> 836468 T4093 C6166 P37740 oasu.PeerSync.sync WARN no frame of reference to tell if we've missed updates
   [junit4]   2> 836469 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=unloadcollection2
   [junit4]   2> 836469 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=unloadcollection2
   [junit4]   2> 836469 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=unloadcollection2
   [junit4]   2> 836470 T4093 C6166 P37740 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 836470 T4093 C6166 P37740 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:57041/nl_eln/unloadcollection1/. core=unloadcollection2
   [junit4]   2> 836471 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C6167 name=unloadcollection1 org.apache.solr.core.SolrCore@1500442 url=http://127.0.0.1:57041/nl_eln/unloadcollection1 node=127.0.0.1:57041_nl_eln C6167_STATE=coll:unloadcollection core:unloadcollection1 props:{state=active, base_url=http://127.0.0.1:57041/nl_eln, core=unloadcollection1, node_name=127.0.0.1:57041_nl_eln, leader=true}
   [junit4]   2> 836475 T4030 C6167 P57041 oasc.SolrCore.execute [unloadcollection1] webapp=/nl_eln path=/get params={wt=javabin&version=2&getVersions=100&distrib=false&qt=/get} status=0 QTime=0 
   [junit4]   2> 836476 T4029 C6167 P57041 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 836477 T4029 C6167 P57041 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 836477 T4029 C6167 P57041 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 836477 T4029 C6167 P57041 oasup.LogUpdateProcessor.finish [unloadcollection1] webapp=/nl_eln path=/update params={wt=javabin&openSearcher=false&version=2&commit_end_point=true&commit=true&softCommit=false&waitSearcher=true} {commit=} 0 1
   [junit4]   2> 836478 T4093 C6166 P37740 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 836478 T4093 C6166 P37740 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 836481 T4028 C6167 P57041 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 836481 T4028 C6167 P57041 oasc.SolrCore.execute [unloadcollection1] webapp=/nl_eln path=/replication params={wt=javabin&version=2&command=indexversion&qt=/replication} status=0 QTime=1 
   [junit4]   2> 836482 T4093 C6166 P37740 oasc.RecoveryStrategy.replay No replay needed. core=unloadcollection2
   [junit4]   2> 836482 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=unloadcollection2
   [junit4]   2> 836482 T4093 C6166 P37740 oasc.ZkController.publish publishing core=unloadcollection2 state=active
   [junit4]   2> 836482 T4093 C6166 P37740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 836483 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 836483 T4093 C6166 P37740 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=unloadcollection2
   [junit4]   2> 836485 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 836485 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37740/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37740_nl_eln",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 836488 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 836488 T4090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836488 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836489 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836489 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836489 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836489 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 836488 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837473 T3989 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 837474 T3989 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: unloadcollection
   [junit4]   2> 837476 T4061 oasc.ZkController.publish publishing core=unloadcollection3 state=down
   [junit4]   2> 837476 T4061 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 837477 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 837477 T4061 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 837479 T4016 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 837479 T4016 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39913/nl_eln",
   [junit4]   2> 	  "core":"unloadcollection3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39913_nl_eln",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"unloadcollection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 837480 T4016 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 837480 T4016 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 837482 T4015 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 837712 T4084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837713 T4015 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837713 T4090 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837713 T4052 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837713 T4021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837713 T4036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 837712 T4068 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838478 T4061 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for unloadcollection3
   [junit4]   2> 838479 T4061 oasc.CoreContainer.create Creating SolrCore 'unloadcollection3' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/unloadcollection3
   [junit4]   2> 838479 T4061 oasc.ZkController.createCollectionZkNode Check for collection zkNode:unloadcollection
   [junit4]   2> 838480 T4061 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 838480 T4061 oascc.ZkStateReader.readConfigName Load collection config from:/collections/unloadcollection
   [junit4]   2> 838482 T4061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.UnloadDistributedZkTest-jetty3-1388903177326/unloadcollection3/'
   [junit4]   2> 838519 T4061 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 838571 T4061 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 838573 T4061 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 838581 T4061 oass.IndexSchema.readSchema [unloadcollection3] Schema name=test
   [junit4]   2> 839006 T4061 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 839012 T4061 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 839015 T4061 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 839027 T4061 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839030 T4061 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839034 T4061 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839035 T4061 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839035 T4061 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected ke

[...truncated too long message...]

4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:276)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 903815 T4084 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 903815 T4084 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 79753 T3988 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 99825 T3988 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 99826 T3988 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 102828 T3988 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {intDefault=Pulsing41(freqCutoff=12 minBlockSize=2 maxBlockSize=12), text=PostingsFormat(name=Lucene41WithOrds), _version_=Pulsing41(freqCutoff=12 minBlockSize=2 maxBlockSize=12), a_t=PostingsFormat(name=FSTPulsing41), range_facet_sl=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=FSTPulsing41), range_facet_l=PostingsFormat(name=FSTPulsing41), other_tl1=Pulsing41(freqCutoff=12 minBlockSize=2 maxBlockSize=12), id=Lucene41(blocksize=128), timestamp=PostingsFormat(name=FSTPulsing41), multiDefault=PostingsFormat(name=FSTPulsing41), a_si=Pulsing41(freqCutoff=12 minBlockSize=2 maxBlockSize=12)}, docValues:{timestamp=DocValuesFormat(name=Memory)}, sim=DefaultSimilarity, locale=es_VE, timezone=Asia/Thimbu
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_45 (32-bit)/cpus=8,threads=2,free=38026792,total=178429952
   [junit4]   2> NOTE: All tests run in this JVM: [MBeansHandlerTest, DistributedTermsComponentTest, TestJoin, ExpressionTest, TestLuceneMatchVersion, CoreAdminHandlerTest, BinaryUpdateRequestHandlerTest, ResourceLoaderTest, SyncSliceTest, SearchHandlerTest, TestBM25SimilarityFactory, QueryParsingTest, EnumFieldTest, TestReplicationHandler, DistributedQueryElevationComponentTest, SliceStateTest, TestWriterPerf, TestQueryUtils, SystemInfoHandlerTest, ZkControllerTest, TestFieldCollectionResource, SolrIndexSplitterTest, TestSystemIdResolver, TestExtendedDismaxParser, LukeRequestHandlerTest, TestPseudoReturnFields, CircularListTest, ReturnFieldsTest, TestValueSourceCache, NoFacetTest, ParsingFieldUpdateProcessorsTest, TestLMDirichletSimilarityFactory, BJQParserTest, TestSort, ResponseLogComponentTest, SuggesterFSTTest, TestDynamicFieldCollectionResource, TestBadConfig, FileBasedSpellCheckerTest, DocValuesTest, QueryElevationComponentTest, TermVectorComponentTest, UpdateParamsTest, OverseerTest, DistributedDebugComponentTest, TestSolrDeletionPolicy1, TermsComponentTest, TestRealTimeGet, TestRemoteStreaming, TestDistributedMissingSort, CacheHeaderTest, LegacyHTMLStripCharFilterTest, BasicFunctionalityTest, TestPHPSerializedResponseWriter, ModifyConfFileTest, TestReloadAndDeleteDocs, TestDocumentBuilder, TestRandomDVFaceting, HdfsDirectoryTest, BasicDistributedZkTest, CoreAdminCreateDiscoverTest, TestShardHandlerFactory, TestConfig, SpatialFilterTest, LeaderElectionTest, AutoCommitTest, TestCollapseQParserPlugin, UUIDFieldTest, TestPerFieldSimilarity, BadComponentTest, TestStressVersions, DeleteReplicaTest, TriLevelCompositeIdRoutingTest, TestMergePolicyConfig, FieldMutatingUpdateProcessorTest, WordBreakSolrSpellCheckerTest, SolrXmlInZkTest, DocumentBuilderTest, SolrCoreTest, UnloadDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=UnloadDistributedZkTest -Dtests.seed=162A6C54E66BCAC3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_VE -Dtests.timezone=Asia/Thimbu -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | UnloadDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.UnloadDistributedZkTest: 
   [junit4]    >    1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([162A6C54E66BCAC3]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=4212, name=Timer-346, state=WAITING, group=TGRP-UnloadDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([162A6C54E66BCAC3]:0)
   [junit4] Completed on J0 in 102.88s, 1 test, 2 errors <<< FAILURES!

[...truncated 548 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 358 suites, 1577 tests, 2 suite-level errors, 805 ignored (4 assumptions)

Total time: 42 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_45 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure