You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/19 21:51:57 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_60-ea-b15) - Build # 10223 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10223/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=4501, name=Thread-1738, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4501, name=Thread-1738, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
	at __randomizedtesting.SeedInfo.seed([4C0EB3CB549DD8C7:CDE83DD323C2B8FB]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not find collection:awholynewstresscollection_collection0_0
	at __randomizedtesting.SeedInfo.seed([4C0EB3CB549DD8C7]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:636)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1006)




Build Log:
[...truncated 11405 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/init-core-data-001
   [junit4]   2> 820737 T4118 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 820737 T4118 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 820741 T4118 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 820741 T4118 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 820742 T4119 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 820842 T4118 oasc.ZkTestServer.run start zk server on port:35739
   [junit4]   2> 820843 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821127 T4125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@165c55f name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821128 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821128 T4118 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 821132 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821134 T4127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69caa18f name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821134 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821134 T4118 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 821139 T4118 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 821141 T4118 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 821142 T4118 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 821143 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 821144 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 821146 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 821146 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 821148 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 821148 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 821150 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 821150 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 821151 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 821151 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 821153 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 821153 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 821154 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 821154 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 821156 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 821156 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 821158 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 821158 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 821159 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 821160 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 821161 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 821161 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 821163 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821164 T4129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104babb8 name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821164 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821165 T4118 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 821169 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821170 T4131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ac5d896 name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821170 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821171 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 821171 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 821173 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 821174 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 821175 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 821176 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 821177 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 821177 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 821179 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 821179 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 821180 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 821180 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 821182 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 821182 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 821183 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 821183 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 821185 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 821185 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 821186 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 821186 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 821188 T4118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 821188 T4118 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 821402 T4118 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 821405 T4118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59666
   [junit4]   2> 821405 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 821406 T4118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 821406 T4118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002
   [junit4]   2> 821406 T4118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/'
   [junit4]   2> 821417 T4118 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solr.xml
   [junit4]   2> 821439 T4118 oasc.CoreContainer.<init> New CoreContainer 378604427
   [junit4]   2> 821439 T4118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/]
   [junit4]   2> 821440 T4118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 821440 T4118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 821440 T4118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 821440 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 821440 T4118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 821441 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 821441 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 821441 T4118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 821441 T4118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 821442 T4118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 821442 T4118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 821442 T4118 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 821442 T4118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35739/solr
   [junit4]   2> 821443 T4118 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 821443 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821445 T4142 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c8cde40 name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821446 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821447 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821448 T4144 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f59ddd3 name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821448 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821449 T4118 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 821451 T4118 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 821453 T4118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 821456 T4118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 821458 T4118 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 821460 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 821461 T4118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59666_
   [junit4]   2> 821461 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59666_
   [junit4]   2> 821463 T4118 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 821464 T4118 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 821467 T4118 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59666_
   [junit4]   2> 821467 T4118 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 821469 T4118 oasc.Overseer.start Overseer (id=91785038045708293-127.0.0.1:59666_-n_0000000000) starting
   [junit4]   2> 821471 T4118 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 821487 T4146 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 821487 T4118 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 821489 T4118 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 821490 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 821491 T4145 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 821494 T4118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 821494 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 821495 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821496 T4149 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d1a75ea name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821496 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821497 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 821498 T4118 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 821709 T4118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 821709 T4118 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 821712 T4118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34418
   [junit4]   2> 821712 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 821712 T4118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 821713 T4118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003
   [junit4]   2> 821713 T4118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/'
   [junit4]   2> 821727 T4118 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solr.xml
   [junit4]   2> 821765 T4118 oasc.CoreContainer.<init> New CoreContainer 1904726985
   [junit4]   2> 821765 T4118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/]
   [junit4]   2> 821766 T4118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 821766 T4118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 821766 T4118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 821767 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 821767 T4118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 821767 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 821767 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 821767 T4118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 821768 T4118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 821768 T4118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 821768 T4118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 821768 T4118 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 821769 T4118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35739/solr
   [junit4]   2> 821769 T4118 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 821770 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821771 T4160 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c616b5a name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821771 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821772 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 821773 T4162 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15848ae name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821774 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 821777 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 822779 T4118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34418_
   [junit4]   2> 822780 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34418_
   [junit4]   2> 822782 T4144 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 822782 T4149 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 822783 T4162 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 822786 T4118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 822786 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 822994 T4118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 822995 T4118 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 822997 T4118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40551
   [junit4]   2> 822998 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 822998 T4118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 822998 T4118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-004
   [junit4]   2> 822999 T4118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-004/'
   [junit4]   2> 823010 T4118 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-004/solr.xml
   [junit4]   2> 823030 T4118 oasc.CoreContainer.<init> New CoreContainer 619972720
   [junit4]   2> 823031 T4118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-004/]
   [junit4]   2> 823031 T4118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 823031 T4118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 823032 T4118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 823032 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 823032 T4118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 823032 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 823032 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 823033 T4118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 823033 T4118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 823033 T4118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 823033 T4118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 823034 T4118 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 823034 T4118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35739/solr
   [junit4]   2> 823034 T4118 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 823035 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 823037 T4174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11560404 name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 823037 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 823039 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 823040 T4176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34d4efe8 name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 823040 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 823044 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 824046 T4118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40551_
   [junit4]   2> 824068 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40551_
   [junit4]   2> 824093 T4144 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 824093 T4176 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 824093 T4162 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 824093 T4149 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 824122 T4118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 824122 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 824330 T4118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 824330 T4118 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 824333 T4118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48356
   [junit4]   2> 824333 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 824334 T4118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 824334 T4118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005
   [junit4]   2> 824334 T4118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/'
   [junit4]   2> 824345 T4118 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solr.xml
   [junit4]   2> 824366 T4118 oasc.CoreContainer.<init> New CoreContainer 29298629
   [junit4]   2> 824366 T4118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/]
   [junit4]   2> 824367 T4118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 824367 T4118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 824367 T4118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 824367 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 824368 T4118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 824368 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 824368 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 824368 T4118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 824368 T4118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 824369 T4118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 824369 T4118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 824369 T4118 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 824370 T4118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35739/solr
   [junit4]   2> 824370 T4118 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 824370 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 824389 T4188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26a44f68 name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 824389 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 824413 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 824429 T4190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3014fb84 name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 824429 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 824530 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 825533 T4118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48356_
   [junit4]   2> 825551 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48356_
   [junit4]   2> 825569 T4176 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 825569 T4190 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 825569 T4162 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 825569 T4144 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 825569 T4149 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 825591 T4118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 825591 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 825796 T4118 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 825796 T4118 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 825799 T4118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47166
   [junit4]   2> 825799 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 825800 T4118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 825800 T4118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006
   [junit4]   2> 825800 T4118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/'
   [junit4]   2> 825811 T4118 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solr.xml
   [junit4]   2> 825831 T4118 oasc.CoreContainer.<init> New CoreContainer 1499901359
   [junit4]   2> 825832 T4118 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/]
   [junit4]   2> 825832 T4118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 825832 T4118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 825833 T4118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 825833 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 825833 T4118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 825833 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 825833 T4118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 825834 T4118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 825834 T4118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 825834 T4118 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 825834 T4118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 825835 T4118 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 825835 T4118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35739/solr
   [junit4]   2> 825835 T4118 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 825836 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825860 T4202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6414ea03 name:ZooKeeperConnection Watcher:127.0.0.1:35739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825861 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 825879 T4118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 825899 T4204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@239a6c6f name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 825899 T4118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 826000 T4118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 827002 T4118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47166_
   [junit4]   2> 827018 T4118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47166_
   [junit4]   2> 827039 T4162 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827039 T4190 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827039 T4204 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827039 T4176 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827039 T4144 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827039 T4149 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 827061 T4118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 827062 T4118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 827069 T4157 oasha.CollectionsHandler.handleCreateAction Creating Collection : collection.configName=conf1&numShards=2&name=solrj_collection&action=CREATE&router.field=myOwnField&version=2&wt=javabin&replicationFactor=2&router.name=compositeId
   [junit4]   2> 827090 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 827109 T4146 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 827109 T4146 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 827110 T4146 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 827110 T4146 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 833709 T4120 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14616017c690006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 833809 T4149 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d1a75ea name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 833810 T4149 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   2> 834967 T4149 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d1a75ea name:ZooKeeperConnection Watcher:127.0.0.1:35739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 838368 T4122 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 11257ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 838369 T4122 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception:  java.nio.channels.CancelledKeyException
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
   [junit4]   2> 	at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
   [junit4]   2> 	at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)
   [junit4]   2> 
   [junit4]   2> 838373 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838374 T4145 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 838375 T4145 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 838375 T4145 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 838378 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838379 T4149 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838379 T4144 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838379 T4190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838380 T4204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838380 T4162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838379 T4176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838473 T4146 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 838473 T4146 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:47166_
   [junit4]   2> 838474 T4146 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:59666_
   [junit4]   2> 838475 T4146 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:34418_
   [junit4]   2> 838476 T4199 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&numShards=2&name=solrj_collection_shard1_replica1&shard=shard1&action=CREATE&version=2&wt=javabin&collection=solrj_collection&qt=/admin/cores
   [junit4]   2> 838477 T4146 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:48356_
   [junit4]   2> 838477 T4139 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&numShards=2&name=solrj_collection_shard1_replica2&shard=shard1&action=CREATE&version=2&wt=javabin&collection=solrj_collection&qt=/admin/cores
   [junit4]   2> 838478 T4199 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 838478 T4156 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&numShards=2&name=solrj_collection_shard2_replica1&shard=shard2&action=CREATE&version=2&wt=javabin&collection=solrj_collection&qt=/admin/cores
   [junit4]   2> 838479 T4186 oasha.CoreAdminHandler.handleCreateAction core create command collection.configName=conf1&numShards=2&name=solrj_collection_shard2_replica2&shard=shard2&action=CREATE&version=2&wt=javabin&collection=solrj_collection&qt=/admin/cores
   [junit4]   2> 838480 T4199 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 838480 T4139 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 838480 T4156 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 838480 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838480 T4186 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 838483 T4139 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 838483 T4156 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 838483 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838483 T4186 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 838483 T4145 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 838484 T4145 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47166",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47166_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 838486 T4145 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59666",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59666_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 838488 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838488 T4145 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:34418",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34418_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 838491 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838491 T4145 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48356",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48356_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 838492 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 838594 T4190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838594 T4149 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838594 T4204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838594 T4176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838594 T4162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 838594 T4144 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 839480 T4199 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 839481 T4199 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solr.xml
   [junit4]   2> 839481 T4199 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 839482 T4199 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 839482 T4199 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839483 T4139 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 839483 T4199 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839483 T4199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 839483 T4156 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 839484 T4186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 839483 T4139 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solr.xml
   [junit4]   2> 839484 T4186 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solr.xml
   [junit4]   2> 839485 T4186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 839484 T4139 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 839484 T4156 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solr.xml
   [junit4]   2> 839485 T4156 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 839485 T4186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 839486 T4139 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 839486 T4156 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 839486 T4156 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839486 T4186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839486 T4139 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839487 T4156 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839487 T4156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 839490 T4186 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839490 T4139 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839490 T4186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/'
   [junit4]   2> 839491 T4139 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/'
   [junit4]   2> 839530 T4199 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 839535 T4186 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 839535 T4156 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 839540 T4139 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 839558 T4186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 839559 T4186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 839565 T4186 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 839570 T4139 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 839570 T4199 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 839570 T4156 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 839571 T4139 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 839572 T4156 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 839572 T4199 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 839581 T4199 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 839581 T4156 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 839581 T4139 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 839893 T4186 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839894 T4186 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839894 T4186 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 839901 T4186 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 839904 T4186 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 839918 T4186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839921 T4199 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839921 T4186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839922 T4199 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839922 T4199 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 839925 T4186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839926 T4199 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 839926 T4186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839927 T4186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839927 T4186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839928 T4186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839928 T4186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839929 T4186 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 839928 T4199 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 839928 T4156 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839930 T4156 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839929 T4139 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839929 T4186 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 839930 T4156 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 839930 T4186 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/data/
   [junit4]   2> 839930 T4139 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 839931 T4186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@632dee4
   [junit4]   2> 839931 T4139 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 839932 T4186 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/data
   [junit4]   2> 839932 T4186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 839933 T4186 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 839933 T4186 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-005/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 839934 T4186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=30.73828125, floorSegmentMB=1.23828125, forceMergeDeletesPctAllowed=25.906265788419717, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6824553551717729
   [junit4]   2> 839935 T4186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@b833da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@305b9b00),segFN=segments_1,generation=1}
   [junit4]   2> 839935 T4186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839936 T4156 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 839937 T4139 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 839937 T4199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839939 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 839939 T4156 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 839940 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 839940 T4199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839940 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 839940 T4139 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 839940 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839941 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839941 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 839942 T4186 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 839942 T4186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 839943 T4186 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 839943 T4199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839943 T4186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 839944 T4186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 839944 T4186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 839944 T4199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839944 T4199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839944 T4186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 839945 T4199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839945 T4186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 839946 T4186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 839946 T4199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839946 T4199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839946 T4186 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 839947 T4199 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 839947 T4199 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 839948 T4199 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 839948 T4199 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@632dee4
   [junit4]   2> 839949 T4199 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/data
   [junit4]   2> 839949 T4199 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 839950 T4199 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 839950 T4199 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-006/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 839950 T4186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 839951 T4199 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=30.73828125, floorSegmentMB=1.23828125, forceMergeDeletesPctAllowed=25.906265788419717, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6824553551717729
   [junit4]   2> 839951 T4139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839951 T4199 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10ff90d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@325444a8),segFN=segments_1,generation=1}
   [junit4]   2> 839951 T4156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839952 T4199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839953 T4186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 839954 T4186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 839955 T4186 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 839955 T4139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839955 T4199 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 839955 T4186 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@b833da lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@305b9b00),segFN=segments_1,generation=1}
   [junit4]   2> 839956 T4186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839955 T4156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 839956 T4186 oass.SolrIndexSearcher.<init> Opening Searcher@62510ca8[solrj_collection_shard2_replica2] main
   [junit4]   2> 839956 T4199 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 839957 T4186 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839957 T4199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 839957 T4199 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839958 T4186 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839958 T4186 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 839958 T4199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839958 T4186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 839959 T4186 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 839958 T4199 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 839959 T4139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839959 T4186 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 839959 T4156 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839960 T4186 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 839959 T4199 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 839961 T4156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839961 T4156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839961 T4199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 839961 T4199 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 839960 T4186 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 839960 T4139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839962 T4139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839962 T4139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839962 T4199 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 839962 T4186 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 839963 T4199 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 839963 T4199 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 839961 T4156 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 839964 T4199 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 839963 T4139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839964 T4199 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 839963 T4186 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 839965 T4199 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 839964 T4156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 839964 T4139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839965 T4156 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 839965 T4199 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 839966 T4156 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 839966 T4139 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 839966 T4156 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 839967 T4139 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 839967 T4199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 839967 T4210 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica2] Registered new searcher Searcher@62510ca8[solrj_collection_shard2_replica2] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 839967 T4156 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 839968 T4186 oasc.CoreContainer.registerCore registering core: solrj_collection_shard2_replica2
   [junit4]   2> 839968 T4156 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@632dee4
   [junit4]   2> 839967 T4139 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/data/
   [junit4]   2> 839968 T4186 oasc.ZkController.register Register replica - core:solrj_collection_shard2_replica2 address:http://127.0.0.1:48356 collection:solrj_collection shard:shard2
   [junit4]   2> 839969 T4139 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@632dee4
   [junit4]   2> 839969 T4156 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 839969 T4199 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 839970 T4139 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/data
   [junit4]   2> 839970 T4199 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 839970 T4186 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard2/election
   [junit4]   2> 839969 T4156 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 839970 T4199 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 839970 T4139 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 839971 T4156 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 839971 T4199 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10ff90d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@325444a8),segFN=segments_1,generation=1}
   [junit4]   2> 839971 T4139 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 839972 T4156 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 839972 T4199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839972 T4156 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=30.73828125, floorSegmentMB=1.23828125, forceMergeDeletesPctAllowed=25.906265788419717, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6824553551717729
   [junit4]   2> 839972 T4139 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-4C0EB3CB549DD8C7-001/tempDir-002/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 839973 T4199 oass.SolrIndexSearcher.<init> Opening Searcher@79a11c68[solrj_collection_shard1_replica1] main
   [junit4]   2> 839973 T4139 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=30.73828125, floorSegmentMB=1.23828125, forceMergeDeletesPctAllowed=25.906265788419717, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6824553551717729
   [junit4]   2> 839974 T4156 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2c17187f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@40502e3a),segFN=segments_1,generation=1}
   [junit4]   2> 839974 T4156 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839974 T4199 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 839974 T4139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2822ae95 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5f149350),segFN=segments_1,generation=1}
   [junit4]   2> 839975 T4186 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 839975 T4139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 839976 T4199 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 839976 T4199 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 839976 T4199 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 839977 T4199 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 839977 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839977 T4199 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 839977 T4186 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=6.045298E14ms
   [junit4]   2> 839978 T4186 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 839978 T4186 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 839977 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839978 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C1054 name=solrj_collection_shard2_replica2 org.apache.solr.core.SolrCore@7c100a13 url=http://127.0.0.1:48356/solrj_collection_shard2_replica2 node=127.0.0.1:48356_ C1054_STATE=coll:solrj_collection core:solrj_collection_shard2_replica2 props:{state=down, base_url=http://127.0.0.1:48356, core=solrj_collection_shard2_replica2, node_name=127.0.0.1:48356_}
   [junit4]   2> 839978 T4186 C1054 P48356 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48356/solrj_collection_shard2_replica2/
   [junit4]   2> 839977 T4199 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 839979 T4186 C1054 P48356 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica2 url=http://127.0.0.1:48356 START replicas=[http://127.0.0.1:34418/solrj_collection_shard2_replica1/] nUpdates=100
   [junit4]   2> 839979 T4145 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 839979 T4139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 839979 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 839981 T4139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 839980 T4186 C1054 P48356 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard2_replica2 url=http://127.0.0.1:48356 DONE.  We have no versions.  sync failed.
   [junit4]   2> 839982 T4186 C1054 P48356 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 839980 T4199 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 839982 T4186 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 839981 T4139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 839983 T4139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839981 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 839983 T4139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 839983 T4144 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 839982 T4186 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48356/solrj_collection_shard2_replica2/ shard2
   [junit4]   2> 839982 T4199 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 839984 T4186 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard2
   [junit4]   2> 839983 T4139 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 839983 T4156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 839984 T4139 oasup.UpdateR

[...truncated too long message...]

62 T4204 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 966762 T4204 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 966762 T4204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 966763 T4204 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 966763 T4204 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 966763 T4204 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 966764 T4204 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/addReplicaColl/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=hr_HR, timezone=America/Dawson
   [junit4]   2> NOTE: Linux 3.8.0-39-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=89197424,total=428343296
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedExpandComponentTest, BasicFunctionalityTest, UnloadDistributedZkTest, TestDocSet, ScriptEngineTest, TestShortCircuitedRequests, DocExpirationUpdateProcessorFactoryTest, CSVRequestHandlerTest, FileBasedSpellCheckerTest, TestFieldSortValues, AutoCommitTest, TestConfig, TestNonNRTOpen, TestCloudManagedSchemaAddField, TestDocumentBuilder, AsyncMigrateRouteKeyTest, BadComponentTest, TestInitQParser, TestManagedResourceStorage, BlockDirectoryTest, QueryElevationComponentTest, SliceStateUpdateTest, StatelessScriptUpdateProcessorFactoryTest, ResourceLoaderTest, UpdateParamsTest, EnumFieldTest, RegexBoostProcessorTest, SchemaVersionSpecificBehaviorTest, CustomCollectionTest, TermVectorComponentTest, DateMathParserTest, DeleteShardTest, DistribCursorPagingTest, SimpleFacetsTest, CopyFieldTest, BinaryUpdateRequestHandlerTest, TestImplicitCoreProperties, ClusterStateTest, UUIDFieldTest, TestJoin, TestExtendedDismaxParser, SOLR749Test, TestTrie, TestFunctionQuery, LukeRequestHandlerTest, SSLMigrationTest, IndexSchemaTest, TestElisionMultitermQuery, ShowFileRequestHandlerTest, DistributedDebugComponentTest, CoreAdminHandlerTest, ParsingFieldUpdateProcessorsTest, TestPHPSerializedResponseWriter, InfoHandlerTest, HdfsLockFactoryTest, SolrCoreTest, PrimitiveFieldTypeTest, QueryParsingTest, SampleTest, SoftAutoCommitTest, TestRealTimeGet, TestNRTOpen, TestMergePolicyConfig, HdfsCollectionsAPIDistributedZkTest, RemoteQueryErrorTest, QueryEqualityTest, ExternalFileFieldSortTest, DistributedQueryElevationComponentTest, BJQParserTest, ClusterStateUpdateTest, BasicDistributedZkTest, TestMaxScoreQueryParser, DistributedQueryComponentOptimizationTest, TestBM25SimilarityFactory, TestPseudoReturnFields, TestPostingsSolrHighlighter, TestLuceneMatchVersion, UpdateRequestProcessorFactoryTest, SolrIndexSplitterTest, DirectUpdateHandlerOptimizeTest, TestSerializedLuceneMatchVersion, TestExceedMaxTermLength, TestPerFieldSimilarity, TestRestManager, SystemInfoHandlerTest, TestSchemaVersionResource, HdfsSyncSliceTest, MBeansHandlerTest, OverseerStatusTest, DistributedQueryComponentCustomSortTest, SynonymTokenizerTest, TestSuggestSpellingConverter, OverseerTest, SolrXmlInZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J0 in 147.21s, 1 test, 1 error <<< FAILURES!

[...truncated 584 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 393 suites, 1637 tests, 1 error, 38 ignored (17 assumptions)

Total time: 56 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure