You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/08/18 03:41:35 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 7058 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/7058/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

1 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Server at http://127.0.0.1:56288/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:56288/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
	at __randomizedtesting.SeedInfo.seed([F1E83370E048266F:700EBD6897174653]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 9486 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 84149 T409 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 84155 T409 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1376788841444
   [junit4]   2> 84156 T409 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 84156 T410 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 84256 T409 oasc.ZkTestServer.run start zk server on port:58372
   [junit4]   2> 84260 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84283 T416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49dac702 name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84283 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84284 T409 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 84299 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84300 T418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b4906ec7 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84301 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84301 T409 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 84304 T409 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 84307 T409 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 84309 T409 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 84312 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 84312 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 84319 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 84320 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 84323 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 84323 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 84329 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 84330 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 84333 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 84334 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 84339 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 84340 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 84346 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 84347 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 84350 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 84350 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 84353 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 84354 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 84356 T409 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 84356 T409 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 84457 T409 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 84464 T409 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54012
   [junit4]   2> 84465 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 84465 T409 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 84466 T409 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648
   [junit4]   2> 84467 T409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/'
   [junit4]   2> 84477 T409 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/solr.xml
   [junit4]   2> 84529 T409 oasc.CoreContainer.<init> New CoreContainer 1340157317
   [junit4]   2> 84530 T409 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/]
   [junit4]   2> 84532 T409 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 84533 T409 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 84534 T409 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 84535 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 84536 T409 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 84536 T409 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 84537 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 84538 T409 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 84539 T409 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 84540 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 84548 T409 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58372/solr
   [junit4]   2> 84549 T409 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 84552 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84553 T429 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@be4e79ef name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84554 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84555 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 84561 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84562 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fa0c7e46 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84562 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84564 T409 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 84566 T409 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 84569 T409 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 84571 T409 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54012_
   [junit4]   2> 84572 T409 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54012_
   [junit4]   2> 84574 T409 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 84578 T409 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 84580 T409 oasc.Overseer.start Overseer (id=90229233513201667-127.0.0.1:54012_-n_0000000000) starting
   [junit4]   2> 84583 T409 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 84586 T433 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 84586 T409 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 84588 T409 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 84590 T409 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 84592 T432 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 84595 T434 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 84596 T434 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 84597 T434 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 86096 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 86096 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54012",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:54012_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 86097 T432 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 86098 T432 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 86103 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 86597 T434 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 86598 T434 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/collection1
   [junit4]   2> 86598 T434 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 86599 T434 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 86600 T434 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 86601 T434 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/collection1/'
   [junit4]   2> 86602 T434 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/collection1/lib/README' to classloader
   [junit4]   2> 86603 T434 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/collection1/lib/classes/' to classloader
   [junit4]   2> 86640 T434 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 86672 T434 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 86673 T434 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 86679 T434 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 87011 T434 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 87016 T434 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 87019 T434 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 87028 T434 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 87032 T434 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 87036 T434 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 87037 T434 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 87037 T434 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 87038 T434 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 87039 T434 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 87039 T434 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 87040 T434 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 87040 T434 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376788841648/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/control/data/
   [junit4]   2> 87041 T434 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@aa75c80d
   [junit4]   2> 87042 T434 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/control/data
   [junit4]   2> 87043 T434 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/control/data/index/
   [junit4]   2> 87043 T434 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 87044 T434 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/control/data/index
   [junit4]   2> 87044 T434 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 87046 T434 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cca63c91 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18f71b9b),segFN=segments_1,generation=1}
   [junit4]   2> 87046 T434 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 87050 T434 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 87050 T434 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 87051 T434 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 87051 T434 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 87052 T434 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 87053 T434 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 87053 T434 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 87054 T434 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 87054 T434 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 87055 T434 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 87056 T434 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 87057 T434 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 87057 T434 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 87058 T434 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 87058 T434 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 87060 T434 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 87064 T434 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 87064 T434 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 87065 T434 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35819413530536426]
   [junit4]   2> 87066 T434 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cca63c91 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@18f71b9b),segFN=segments_1,generation=1}
   [junit4]   2> 87067 T434 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 87067 T434 oass.SolrIndexSearcher.<init> Opening Searcher@200dac5d main
   [junit4]   2> 87071 T435 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@200dac5d main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 87072 T434 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 87072 T434 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54012 collection:control_collection shard:shard1
   [junit4]   2> 87073 T434 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 87078 T434 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 87094 T434 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 87096 T434 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 87096 T434 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 87097 T434 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54012/collection1/
   [junit4]   2> 87097 T434 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 87097 T434 oasc.SyncStrategy.syncToMe http://127.0.0.1:54012/collection1/ has no replicas
   [junit4]   2> 87098 T434 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54012/collection1/ shard1
   [junit4]   2> 87098 T434 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 87606 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 87612 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 87654 T434 oasc.ZkController.register We are http://127.0.0.1:54012/collection1/ and leader is http://127.0.0.1:54012/collection1/
   [junit4]   2> 87655 T434 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54012
   [junit4]   2> 87656 T434 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 87657 T434 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 87657 T434 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 87660 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 87662 T409 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 87664 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 87665 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 87675 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 87678 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87680 T438 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@581257d5 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87681 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87683 T409 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 87688 T409 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 87779 T409 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 87781 T409 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45379
   [junit4]   2> 87782 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 87783 T409 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 87783 T409 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978
   [junit4]   2> 87784 T409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/'
   [junit4]   2> 87793 T409 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/solr.xml
   [junit4]   2> 87825 T409 oasc.CoreContainer.<init> New CoreContainer -363214043
   [junit4]   2> 87825 T409 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/]
   [junit4]   2> 87827 T409 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 87827 T409 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 87828 T409 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 87828 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 87829 T409 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 87830 T409 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 87830 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 87831 T409 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 87831 T409 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 87832 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 87837 T409 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58372/solr
   [junit4]   2> 87837 T409 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 87838 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87840 T449 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de0c7bd6 name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87840 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87842 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 87846 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 87847 T451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f338c799 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 87848 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 87851 T409 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 88853 T409 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45379_
   [junit4]   2> 88857 T409 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45379_
   [junit4]   2> 88863 T451 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 88863 T438 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 88863 T431 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 88864 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 88869 T452 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 88870 T452 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 88872 T452 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 89115 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 89116 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54012",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:54012_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 89120 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45379",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:45379_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 89120 T432 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 89121 T432 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 89124 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 89124 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 89124 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 89872 T452 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 89873 T452 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/collection1
   [junit4]   2> 89874 T452 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 89876 T452 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 89877 T452 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 89879 T452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/collection1/'
   [junit4]   2> 89881 T452 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/collection1/lib/README' to classloader
   [junit4]   2> 89882 T452 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/collection1/lib/classes/' to classloader
   [junit4]   2> 89917 T452 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 89949 T452 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 89951 T452 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 89957 T452 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 90287 T452 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 90293 T452 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 90296 T452 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 90307 T452 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 90311 T452 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 90314 T452 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 90316 T452 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 90316 T452 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 90316 T452 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 90317 T452 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 90318 T452 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 90318 T452 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 90318 T452 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376788844978/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty1/
   [junit4]   2> 90319 T452 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@aa75c80d
   [junit4]   2> 90320 T452 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty1
   [junit4]   2> 90320 T452 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty1/index/
   [junit4]   2> 90320 T452 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 90321 T452 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty1/index
   [junit4]   2> 90321 T452 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 90322 T452 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fd5f09ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@aae49733),segFN=segments_1,generation=1}
   [junit4]   2> 90322 T452 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 90325 T452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 90326 T452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 90326 T452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 90326 T452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 90327 T452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 90327 T452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 90328 T452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 90328 T452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 90328 T452 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 90329 T452 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 90330 T452 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 90330 T452 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 90330 T452 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 90331 T452 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 90331 T452 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 90333 T452 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 90335 T452 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 90335 T452 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 90336 T452 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35819413530536426]
   [junit4]   2> 90337 T452 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fd5f09ca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@aae49733),segFN=segments_1,generation=1}
   [junit4]   2> 90337 T452 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 90338 T452 oass.SolrIndexSearcher.<init> Opening Searcher@7d95f374 main
   [junit4]   2> 90342 T453 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d95f374 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 90343 T452 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 90343 T452 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45379 collection:collection1 shard:shard1
   [junit4]   2> 90344 T452 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 90348 T452 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 90353 T452 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 90355 T452 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 90356 T452 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 90356 T452 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45379/collection1/
   [junit4]   2> 90356 T452 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 90357 T452 oasc.SyncStrategy.syncToMe http://127.0.0.1:45379/collection1/ has no replicas
   [junit4]   2> 90357 T452 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45379/collection1/ shard1
   [junit4]   2> 90357 T452 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 90629 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 90635 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 90635 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 90635 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 90663 T452 oasc.ZkController.register We are http://127.0.0.1:45379/collection1/ and leader is http://127.0.0.1:45379/collection1/
   [junit4]   2> 90663 T452 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45379
   [junit4]   2> 90664 T452 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 90665 T452 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 90665 T452 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 90668 T452 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 90669 T409 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 90670 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 90672 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 90766 T409 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 90768 T409 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56288
   [junit4]   2> 90769 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 90770 T409 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 90770 T409 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970
   [junit4]   2> 90771 T409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/'
   [junit4]   2> 90781 T409 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/solr.xml
   [junit4]   2> 90814 T409 oasc.CoreContainer.<init> New CoreContainer 1952438144
   [junit4]   2> 90815 T409 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/]
   [junit4]   2> 90816 T409 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 90817 T409 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 90817 T409 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 90818 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 90819 T409 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 90819 T409 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 90820 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 90820 T409 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 90821 T409 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 90821 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 90826 T409 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58372/solr
   [junit4]   2> 90827 T409 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 90828 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 90830 T465 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ca2363 name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 90830 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 90831 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 90837 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 90838 T467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22a29100 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 90838 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 90841 T409 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 91844 T409 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56288_
   [junit4]   2> 91845 T409 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56288_
   [junit4]   2> 91848 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 91848 T467 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 91848 T431 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 91848 T451 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 91849 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 91849 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 91850 T438 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 91854 T468 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 91854 T468 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 91856 T468 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 92141 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 92142 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45379",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:45379_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 92146 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56288",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:56288_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 92146 T432 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 92146 T432 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 92151 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92151 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92151 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92151 T467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 92856 T468 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 92856 T468 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/collection1
   [junit4]   2> 92857 T468 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 92858 T468 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 92858 T468 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 92859 T468 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/collection1/'
   [junit4]   2> 92860 T468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/collection1/lib/README' to classloader
   [junit4]   2> 92861 T468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/collection1/lib/classes/' to classloader
   [junit4]   2> 92907 T468 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 92959 T468 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 92960 T468 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 92970 T468 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 93336 T468 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 93342 T468 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 93345 T468 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 93357 T468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 93362 T468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 93367 T468 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 93368 T468 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 93368 T468 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 93368 T468 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 93369 T468 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 93370 T468 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 93370 T468 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 93370 T468 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376788847970/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty2/
   [junit4]   2> 93371 T468 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@aa75c80d
   [junit4]   2> 93372 T468 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty2
   [junit4]   2> 93372 T468 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty2/index/
   [junit4]   2> 93372 T468 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 93373 T468 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty2/index
   [junit4]   2> 93373 T468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 93374 T468 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@90e03861 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de4e61b4),segFN=segments_1,generation=1}
   [junit4]   2> 93375 T468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 93379 T468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 93379 T468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 93380 T468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 93380 T468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 93380 T468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 93381 T468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 93381 T468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 93382 T468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 93382 T468 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 93383 T468 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 93383 T468 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 93383 T468 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 93384 T468 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 93384 T468 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 93385 T468 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 93386 T468 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 93389 T468 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 93389 T468 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 93390 T468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35819413530536426]
   [junit4]   2> 93390 T468 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@90e03861 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de4e61b4),segFN=segments_1,generation=1}
   [junit4]   2> 93391 T468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 93391 T468 oass.SolrIndexSearcher.<init> Opening Searcher@ecb5a3d8 main
   [junit4]   2> 93395 T469 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ecb5a3d8 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 93397 T468 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 93397 T468 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56288 collection:collection1 shard:shard2
   [junit4]   2> 93397 T468 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 93402 T468 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 93407 T468 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 93409 T468 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 93409 T468 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 93410 T468 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56288/collection1/
   [junit4]   2> 93410 T468 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 93410 T468 oasc.SyncStrategy.syncToMe http://127.0.0.1:56288/collection1/ has no replicas
   [junit4]   2> 93411 T468 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56288/collection1/ shard2
   [junit4]   2> 93411 T468 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 93655 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 93712 T467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 93712 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 93712 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 93712 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 93715 T468 oasc.ZkController.register We are http://127.0.0.1:56288/collection1/ and leader is http://127.0.0.1:56288/collection1/
   [junit4]   2> 93716 T468 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56288
   [junit4]   2> 93716 T468 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 93716 T468 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 93717 T468 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 93719 T468 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 93720 T409 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 93720 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 93721 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 93799 T409 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 93800 T409 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41509
   [junit4]   2> 93801 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 93802 T409 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 93802 T409 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014
   [junit4]   2> 93803 T409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/'
   [junit4]   2> 93812 T409 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/solr.xml
   [junit4]   2> 93846 T409 oasc.CoreContainer.<init> New CoreContainer 1535354789
   [junit4]   2> 93846 T409 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/]
   [junit4]   2> 93848 T409 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 93848 T409 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 93849 T409 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 93849 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 93850 T409 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 93850 T409 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 93851 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 93852 T409 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 93852 T409 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 93853 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 93857 T409 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58372/solr
   [junit4]   2> 93858 T409 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 93859 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93860 T481 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9845a78e name:ZooKeeperConnection Watcher:127.0.0.1:58372 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93861 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93862 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 93868 T409 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93870 T483 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@278c6997 name:ZooKeeperConnection Watcher:127.0.0.1:58372/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93870 T409 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93873 T409 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 94876 T409 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41509_
   [junit4]   2> 94877 T409 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41509_
   [junit4]   2> 94880 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 94881 T467 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 94881 T431 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 94881 T467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 94881 T483 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 94881 T451 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 94883 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 94881 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 94882 T438 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 94888 T484 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 94888 T484 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 94889 T484 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 95218 T432 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 95218 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56288",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:56288_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 95221 T432 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41509",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "node_name":"127.0.0.1:41509_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 95221 T432 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 95221 T432 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 95262 T451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95262 T467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95262 T431 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95262 T438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95262 T483 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95890 T484 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 95890 T484 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/collection1
   [junit4]   2> 95890 T484 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 95891 T484 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 95892 T484 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 95893 T484 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/collection1/'
   [junit4]   2> 95893 T484 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/collection1/lib/README' to classloader
   [junit4]   2> 95894 T484 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/collection1/lib/classes/' to classloader
   [junit4]   2> 95924 T484 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 95956 T484 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 95958 T484 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 95973 T484 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 96392 T484 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 96398 T484 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 96401 T484 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 96414 T484 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 96418 T484 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 96422 T484 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 96423 T484 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 96424 T484 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 96424 T484 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 96425 T484 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 96426 T484 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 96426 T484 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 96426 T484 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1376788851014/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty3/
   [junit4]   2> 96427 T484 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@aa75c80d
   [junit4]   2> 96428 T484 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty3
   [junit4]   2> 96428 T484 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty3/index/
   [junit4]   2> 96429 T484 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 96430 T484 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376788841444/jetty3/index
   [junit4]   2> 96430 T484 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 96432 T484 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@676a3290 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@649f242b),segFN=segments_1,generation=1}
   [junit4]   2> 96432 T484 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96437 T484 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 96438 T484 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 96438 T484 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 96439 T484 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 96439 T484 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 96439 T484 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 96440 T484 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 96440 T484 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 96441 T484 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 96441 T484 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 96442 T484 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 96443 T484 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 96443 T484 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 96444 T484 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 96444 T484 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 96446 T484 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 96448 T484 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 96448 T484 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 96449 T484 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.35819413530536426]
   [junit4]   2> 96450 T484 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@676a3290 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@649f242b),segFN=segments_1,generation=1}
   [junit4]   2> 96450 T484 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 96451 T484 oass.SolrIndexSearcher.<init> Opening Searcher@a6da0142 main
   [junit4]   2> 96456 T485 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a6da0142 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 96458 T484 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 96459 T484 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41509 collection:collection1 shard:shard1
   [junit4]   2> 96459 T484 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 96465 T484 oasc.ZkController.register We are http://127.0.0.1:41509/collection1/ and leader is http://127.0.0.1:45379/collection1/
   [junit4]   2> 96466 T484 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41509
   [junit4]   2> 96466 T484 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 96466 T484 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 96467 T484 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C109 name=collection1 org.apache.solr.core.SolrCore@c128163e url=http://127.0.0.1:41509/collection1 node=127.0.0.1:41509_ C109_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:41509, state=down, node_name=127.0.0.1:41509_, core=collection1}
   [junit4]   2> 96467 T486 C109 P41509 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 96468 T409 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 96469 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 96468 T486 C109 P41509 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 96470 T409 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 96470 T486 C109 P41509 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 96471 T486 C109 P41509 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 96475 T486 C109 P41509 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 96484 T446 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 96559 T409 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 96562 T409 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47585
   [junit4]   2> 96563 T409 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 96563 T409 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 96564 T409 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1376788853764
   [junit4]   2> 96565 T409 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1376788853764/'
   [junit4]   2> 96573 T409 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1376788853764/solr.xml
   [junit4]   2> 96606 T409 oasc.CoreContainer.<init> New CoreContainer -18667400
   [junit4]   2> 96606 T409 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1376788853764/]
   [junit4]   2> 96608 T409 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 96608 T409 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 96609 T409 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 96610 T409 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 96610 T409 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 96611 T409 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4

[...truncated too long message...]

s/collection1/leaders (2)
   [junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "base_url":"http://127.0.0.1:45379",
   [junit4]   1>            "node_name":"127.0.0.1:45379_",
   [junit4]   1>            "core":"collection1"}
   [junit4]   1>      /solr/collections/collection1/leaders/shard2 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "base_url":"http://127.0.0.1:56288",
   [junit4]   1>            "node_name":"127.0.0.1:56288_",
   [junit4]   1>            "core":"collection1"}
   [junit4]   1>     /solr/collections/collection1/shards (0)
   [junit4]   1>    /solr/collections/onenodecollection (2)
   [junit4]   1>    DATA:
   [junit4]   1>        {
   [junit4]   1>          "router":"compositeId",
   [junit4]   1>          "configName":"conf1"}
   [junit4]   1>     /solr/collections/onenodecollection/leader_elect (1)
   [junit4]   1>      /solr/collections/onenodecollection/leader_elect/shard1 (1)
   [junit4]   1>       /solr/collections/onenodecollection/leader_elect/shard1/election (1)
   [junit4]   1>        /solr/collections/onenodecollection/leader_elect/shard1/election/90229233513201670-core_node1-n_0000000000 (0)
   [junit4]   1>     /solr/collections/onenodecollection/leaders (1)
   [junit4]   1>      /solr/collections/onenodecollection/leaders/shard1 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "base_url":"http://127.0.0.1:45379",
   [junit4]   1>            "node_name":"127.0.0.1:45379_",
   [junit4]   1>            "core":"onenodecollectioncore"}
   [junit4]   1>   /solr/aliases.json (0)
   [junit4]   1>   /solr/live_nodes (5)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:45379_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:54012_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:41509_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:56288_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:47585_ (0)
   [junit4]   1>   /solr/clusterstate.json (0)
   [junit4]   1>   DATA:
   [junit4]   1>       {
   [junit4]   1>         "control_collection":{
   [junit4]   1>           "shards":{"shard1":{
   [junit4]   1>               "range":"80000000-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:54012",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:54012_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "onenodecollection":{
   [junit4]   1>           "shards":{"shard1":{
   [junit4]   1>               "range":"80000000-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:45379",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:45379_",
   [junit4]   1>                   "roles":"none",
   [junit4]   1>                   "core":"onenodecollectioncore",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "collection1":{
   [junit4]   1>           "shards":{
   [junit4]   1>             "shard1":{
   [junit4]   1>               "range":"80000000-ffffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:45379",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:45379_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node3":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:41509",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:41509_",
   [junit4]   1>                   "core":"collection1"}}},
   [junit4]   1>             "shard2":{
   [junit4]   1>               "range":"0-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node2":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:56288",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:56288_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node4":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:47585",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:47585_",
   [junit4]   1>                   "core":"collection1"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"}}
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=F1E83370E048266F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pl_PL -Dtests.timezone=Asia/Ujung_Pandang -Dtests.file.encoding=UTF-8
   [junit4] ERROR   28.2s J1 | BasicDistributedZk2Test.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:56288/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F1E83370E048266F:700EBD6897174653]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 112325 T409 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 28185 T408 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 113206 T500 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 113207 T500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 113207 T500 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=105), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=105)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=pl_PL, timezone=Asia/Ujung_Pandang
   [junit4]   2> NOTE: Linux 3.2.0-51-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=3,free=103972816,total=180355072
   [junit4]   2> NOTE: All tests run in this JVM: [TestUpdate, SolrPluginUtilsTest, IndexSchemaRuntimeFieldTest, CustomCollectionTest, DirectSolrSpellCheckerTest, TestIndexingPerformance, BasicDistributedZk2Test]
   [junit4] Completed on J1 in 29.21s, 1 test, 1 error <<< FAILURES!

[...truncated 925 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:382: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 322 suites, 1372 tests, 1 error, 34 ignored (6 assumptions)

Total time: 42 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure