You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/08/29 14:54:53 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 7221 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/7221/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

1 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Server at http://127.0.0.1:35466/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:35466/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
	at __randomizedtesting.SeedInfo.seed([8B26208A68D23A8C:AC0AE921F8D5AB0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 9830 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 768079 T2172 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 768083 T2172 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1377780316085
   [junit4]   2> 768084 T2172 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 768085 T2173 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 768185 T2172 oasc.ZkTestServer.run start zk server on port:53604
   [junit4]   2> 768186 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 768544 T2179 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4632d7d0 name:ZooKeeperConnection Watcher:127.0.0.1:53604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 768544 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 768545 T2172 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 768549 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 768550 T2181 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2eb27f6 name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 768550 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 768551 T2172 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 768554 T2172 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 768556 T2172 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 768557 T2172 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 768559 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 768560 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 768563 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 768564 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 768567 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 768567 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 768569 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 768570 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 768572 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 768573 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 768575 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 768576 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 768578 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 768579 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 768581 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 768581 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 768583 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 768584 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 768586 T2172 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 768587 T2172 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 768661 T2172 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 768665 T2172 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39818
   [junit4]   2> 768666 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 768666 T2172 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 768667 T2172 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590
   [junit4]   2> 768667 T2172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/'
   [junit4]   2> 768675 T2172 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/solr.xml
   [junit4]   2> 768707 T2172 oasc.CoreContainer.<init> New CoreContainer -936870103
   [junit4]   2> 768708 T2172 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/]
   [junit4]   2> 768709 T2172 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 768710 T2172 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 768711 T2172 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 768711 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 768712 T2172 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 768713 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 768713 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 768714 T2172 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 768715 T2172 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 768715 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 768720 T2172 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 768721 T2172 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 768722 T2172 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53604/solr
   [junit4]   2> 768722 T2172 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 768723 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 768725 T2192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46503358 name:ZooKeeperConnection Watcher:127.0.0.1:53604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 768725 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 768727 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 768731 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 768733 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fc4c8a24 name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 768733 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 768734 T2172 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 768737 T2172 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 768739 T2172 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 768740 T2172 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39818_
   [junit4]   2> 768741 T2172 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39818_
   [junit4]   2> 768744 T2172 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 768747 T2172 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 768749 T2172 oasc.Overseer.start Overseer (id=90294210795143171-127.0.0.1:39818_-n_0000000000) starting
   [junit4]   2> 768751 T2172 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 768755 T2196 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 768755 T2172 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 768758 T2172 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 768759 T2172 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 768762 T2195 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 768765 T2197 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 768765 T2197 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 768766 T2197 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 770264 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 770265 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39818",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:39818_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 770265 T2195 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 770266 T2195 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 770269 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 770766 T2197 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 770767 T2197 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/collection1
   [junit4]   2> 770767 T2197 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 770768 T2197 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 770768 T2197 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 770769 T2197 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/collection1/'
   [junit4]   2> 770770 T2197 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/collection1/lib/classes/' to classloader
   [junit4]   2> 770770 T2197 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/collection1/lib/README' to classloader
   [junit4]   2> 770807 T2197 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 770843 T2197 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 770844 T2197 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 770851 T2197 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 771244 T2197 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 771251 T2197 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 771254 T2197 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 771266 T2197 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 771270 T2197 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 771274 T2197 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 771275 T2197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 771275 T2197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 771275 T2197 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 771276 T2197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 771277 T2197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 771277 T2197 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 771278 T2197 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1377780316590/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/control/data/
   [junit4]   2> 771278 T2197 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@438b213b
   [junit4]   2> 771279 T2197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/control/data
   [junit4]   2> 771279 T2197 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/control/data/index/
   [junit4]   2> 771279 T2197 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 771280 T2197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/control/data/index
   [junit4]   2> 771280 T2197 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=646070326, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 771281 T2197 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@91a9efda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b258d39),segFN=segments_1,generation=1}
   [junit4]   2> 771282 T2197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 771283 T2197 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 771284 T2197 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 771284 T2197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 771285 T2197 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 771285 T2197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 771285 T2197 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 771286 T2197 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 771286 T2197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 771287 T2197 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 771287 T2197 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 771287 T2197 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 771288 T2197 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 771288 T2197 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 771289 T2197 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 771289 T2197 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 771290 T2197 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 771293 T2197 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 771293 T2197 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 771294 T2197 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=839210927, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 771295 T2197 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@91a9efda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3b258d39),segFN=segments_1,generation=1}
   [junit4]   2> 771295 T2197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 771296 T2197 oass.SolrIndexSearcher.<init> Opening Searcher@192af076 main
   [junit4]   2> 771297 T2198 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@192af076 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 771298 T2197 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 771298 T2197 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39818 collection:control_collection shard:shard1
   [junit4]   2> 771298 T2197 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 771303 T2197 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 771308 T2197 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 771310 T2197 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 771311 T2197 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 771311 T2197 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39818/collection1/
   [junit4]   2> 771311 T2197 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 771312 T2197 oasc.SyncStrategy.syncToMe http://127.0.0.1:39818/collection1/ has no replicas
   [junit4]   2> 771312 T2197 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39818/collection1/ shard1
   [junit4]   2> 771312 T2197 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 771772 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 771779 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 771818 T2197 oasc.ZkController.register We are http://127.0.0.1:39818/collection1/ and leader is http://127.0.0.1:39818/collection1/
   [junit4]   2> 771819 T2197 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39818
   [junit4]   2> 771819 T2197 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 771819 T2197 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 771820 T2197 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 771821 T2197 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 771822 T2172 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 771823 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 771823 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 771828 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 771829 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 771830 T2201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c167a0 name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 771830 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 771831 T2172 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 771833 T2172 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 771899 T2172 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 771901 T2172 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45466
   [junit4]   2> 771901 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 771902 T2172 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 771902 T2172 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835
   [junit4]   2> 771903 T2172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/'
   [junit4]   2> 771910 T2172 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/solr.xml
   [junit4]   2> 771941 T2172 oasc.CoreContainer.<init> New CoreContainer 848518794
   [junit4]   2> 771942 T2172 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/]
   [junit4]   2> 771943 T2172 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 771943 T2172 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 771944 T2172 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 771945 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 771945 T2172 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 771946 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 771946 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 771947 T2172 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 771947 T2172 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 771948 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 771952 T2172 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 771953 T2172 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 771954 T2172 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53604/solr
   [junit4]   2> 771954 T2172 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 771955 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 771956 T2212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e753ed12 name:ZooKeeperConnection Watcher:127.0.0.1:53604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 771957 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 771958 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 771963 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 771964 T2214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63c42c9b name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 771964 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 771967 T2172 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 772970 T2172 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45466_
   [junit4]   2> 772971 T2172 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45466_
   [junit4]   2> 772974 T2201 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 772974 T2194 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 772974 T2214 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 772975 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 772979 T2215 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 772979 T2215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 772980 T2215 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 773283 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 773284 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39818",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:39818_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 773287 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45466",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:45466_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 773288 T2195 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 773288 T2195 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 773291 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 773291 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 773291 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 773980 T2215 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 773981 T2215 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/collection1
   [junit4]   2> 773981 T2215 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 773982 T2215 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 773982 T2215 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 773983 T2215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/collection1/'
   [junit4]   2> 773984 T2215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/collection1/lib/classes/' to classloader
   [junit4]   2> 773984 T2215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/collection1/lib/README' to classloader
   [junit4]   2> 774014 T2215 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 774050 T2215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 774051 T2215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 774058 T2215 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 774419 T2215 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 774425 T2215 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 774428 T2215 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 774437 T2215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 774441 T2215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 774445 T2215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 774445 T2215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 774446 T2215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 774446 T2215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 774447 T2215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 774447 T2215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 774448 T2215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 774448 T2215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1377780319835/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty1/
   [junit4]   2> 774449 T2215 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@438b213b
   [junit4]   2> 774449 T2215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty1
   [junit4]   2> 774450 T2215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty1/index/
   [junit4]   2> 774450 T2215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 774451 T2215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty1/index
   [junit4]   2> 774451 T2215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=646070326, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 774452 T2215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cfc2a327 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e04daa8b),segFN=segments_1,generation=1}
   [junit4]   2> 774452 T2215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 774455 T2215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 774455 T2215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 774455 T2215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 774456 T2215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 774456 T2215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 774456 T2215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 774457 T2215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 774457 T2215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 774458 T2215 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 774458 T2215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 774459 T2215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 774459 T2215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 774460 T2215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 774460 T2215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 774460 T2215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 774462 T2215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 774464 T2215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 774465 T2215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 774465 T2215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=839210927, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 774466 T2215 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@cfc2a327 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@e04daa8b),segFN=segments_1,generation=1}
   [junit4]   2> 774467 T2215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 774467 T2215 oass.SolrIndexSearcher.<init> Opening Searcher@33114047 main
   [junit4]   2> 774470 T2216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33114047 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 774471 T2215 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 774472 T2215 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45466 collection:collection1 shard:shard1
   [junit4]   2> 774472 T2215 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 774476 T2215 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 774481 T2215 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 774482 T2215 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 774483 T2215 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 774483 T2215 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45466/collection1/
   [junit4]   2> 774483 T2215 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 774484 T2215 oasc.SyncStrategy.syncToMe http://127.0.0.1:45466/collection1/ has no replicas
   [junit4]   2> 774484 T2215 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45466/collection1/ shard1
   [junit4]   2> 774484 T2215 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 774796 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 774802 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 774802 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 774802 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 774840 T2215 oasc.ZkController.register We are http://127.0.0.1:45466/collection1/ and leader is http://127.0.0.1:45466/collection1/
   [junit4]   2> 774840 T2215 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45466
   [junit4]   2> 774840 T2215 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 774841 T2215 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 774841 T2215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 774842 T2215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 774843 T2172 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 774844 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 774845 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 774914 T2172 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 774916 T2172 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35466
   [junit4]   2> 774917 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 774917 T2172 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 774918 T2172 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850
   [junit4]   2> 774918 T2172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/'
   [junit4]   2> 774926 T2172 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/solr.xml
   [junit4]   2> 774956 T2172 oasc.CoreContainer.<init> New CoreContainer 1924047583
   [junit4]   2> 774957 T2172 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/]
   [junit4]   2> 774958 T2172 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 774959 T2172 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 774959 T2172 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 774960 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 774961 T2172 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 774961 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 774962 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 774962 T2172 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 774963 T2172 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 774963 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 774967 T2172 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 774968 T2172 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 774969 T2172 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53604/solr
   [junit4]   2> 774969 T2172 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 774970 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 774972 T2228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@86fc8d05 name:ZooKeeperConnection Watcher:127.0.0.1:53604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 774972 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 774974 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 774979 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 774980 T2230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae2cbf3a name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 774980 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 774983 T2172 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 775986 T2172 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35466_
   [junit4]   2> 775988 T2172 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35466_
   [junit4]   2> 775990 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 775990 T2230 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 775990 T2194 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 775990 T2201 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 775991 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 775991 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 775992 T2214 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 775996 T2231 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 775996 T2231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 775997 T2231 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 776306 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 776307 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45466",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:45466_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 776310 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35466",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:35466_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 776310 T2195 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 776310 T2195 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 776315 T2230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 776315 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 776315 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 776315 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 776998 T2231 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 776998 T2231 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/collection1
   [junit4]   2> 776999 T2231 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 777000 T2231 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 777000 T2231 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 777001 T2231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/collection1/'
   [junit4]   2> 777002 T2231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/collection1/lib/classes/' to classloader
   [junit4]   2> 777003 T2231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/collection1/lib/README' to classloader
   [junit4]   2> 777047 T2231 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 777083 T2231 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 777084 T2231 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 777090 T2231 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 777444 T2231 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 777449 T2231 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 777452 T2231 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 777461 T2231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 777465 T2231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 777469 T2231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 777470 T2231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 777471 T2231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 777471 T2231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 777472 T2231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 777472 T2231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 777473 T2231 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 777473 T2231 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1377780322850/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty2/
   [junit4]   2> 777474 T2231 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@438b213b
   [junit4]   2> 777475 T2231 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty2
   [junit4]   2> 777475 T2231 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty2/index/
   [junit4]   2> 777476 T2231 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 777476 T2231 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty2/index
   [junit4]   2> 777477 T2231 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=646070326, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 777478 T2231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9ce968a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cf890bc1),segFN=segments_1,generation=1}
   [junit4]   2> 777478 T2231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 777481 T2231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 777481 T2231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 777482 T2231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 777482 T2231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 777483 T2231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 777483 T2231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 777484 T2231 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 777484 T2231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 777484 T2231 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 777485 T2231 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 777486 T2231 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 777486 T2231 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 777486 T2231 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 777487 T2231 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 777488 T2231 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 777489 T2231 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 777493 T2231 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 777493 T2231 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 777494 T2231 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=839210927, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 777495 T2231 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9ce968a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cf890bc1),segFN=segments_1,generation=1}
   [junit4]   2> 777496 T2231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 777497 T2231 oass.SolrIndexSearcher.<init> Opening Searcher@d4c3934a main
   [junit4]   2> 777501 T2232 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d4c3934a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 777503 T2231 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 777504 T2231 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35466 collection:collection1 shard:shard2
   [junit4]   2> 777504 T2231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 777512 T2231 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 777516 T2231 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 777518 T2231 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 777518 T2231 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 777518 T2231 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35466/collection1/
   [junit4]   2> 777519 T2231 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 777519 T2231 oasc.SyncStrategy.syncToMe http://127.0.0.1:35466/collection1/ has no replicas
   [junit4]   2> 777519 T2231 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35466/collection1/ shard2
   [junit4]   2> 777520 T2231 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 777821 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 777827 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 777827 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 777827 T2230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 777827 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 777874 T2231 oasc.ZkController.register We are http://127.0.0.1:35466/collection1/ and leader is http://127.0.0.1:35466/collection1/
   [junit4]   2> 777874 T2231 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35466
   [junit4]   2> 777875 T2231 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 777875 T2231 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 777875 T2231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 777877 T2231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 777879 T2172 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 777879 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 777880 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 777951 T2172 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 777953 T2172 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53973
   [junit4]   2> 777954 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 777954 T2172 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 777955 T2172 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886
   [junit4]   2> 777956 T2172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/'
   [junit4]   2> 777963 T2172 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/solr.xml
   [junit4]   2> 777996 T2172 oasc.CoreContainer.<init> New CoreContainer 537875294
   [junit4]   2> 777997 T2172 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/]
   [junit4]   2> 777998 T2172 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 777998 T2172 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 777999 T2172 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 777999 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 778000 T2172 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 778001 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 778001 T2172 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 778002 T2172 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 778002 T2172 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 778003 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 778008 T2172 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 778008 T2172 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 778009 T2172 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53604/solr
   [junit4]   2> 778010 T2172 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 778011 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778012 T2244 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74442084 name:ZooKeeperConnection Watcher:127.0.0.1:53604 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778012 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778015 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 778020 T2172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 778022 T2246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@acf4b563 name:ZooKeeperConnection Watcher:127.0.0.1:53604/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 778022 T2172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 778025 T2172 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 779028 T2172 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53973_
   [junit4]   2> 779029 T2172 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53973_
   [junit4]   2> 779031 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 779032 T2230 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 779032 T2201 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 779032 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 779032 T2246 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 779032 T2230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779032 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779033 T2214 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 779033 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779038 T2247 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 779038 T2247 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 779039 T2247 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 779332 T2195 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 779333 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35466",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:35466_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 779336 T2195 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53973",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "node_name":"127.0.0.1:53973_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 779336 T2195 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 779336 T2195 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 779340 T2246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779340 T2230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779340 T2214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779340 T2201 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 779340 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 780039 T2247 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 780040 T2247 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/collection1
   [junit4]   2> 780040 T2247 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 780041 T2247 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 780041 T2247 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 780042 T2247 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/collection1/'
   [junit4]   2> 780043 T2247 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/collection1/lib/classes/' to classloader
   [junit4]   2> 780044 T2247 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/collection1/lib/README' to classloader
   [junit4]   2> 780077 T2247 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 780112 T2247 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 780113 T2247 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 780121 T2247 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 780492 T2247 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 780498 T2247 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 780500 T2247 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 780509 T2247 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 780513 T2247 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 780517 T2247 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 780518 T2247 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 780518 T2247 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 780519 T2247 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 780520 T2247 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 780520 T2247 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 780520 T2247 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 780521 T2247 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1377780325886/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty3/
   [junit4]   2> 780521 T2247 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@438b213b
   [junit4]   2> 780522 T2247 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty3
   [junit4]   2> 780522 T2247 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty3/index/
   [junit4]   2> 780523 T2247 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 780523 T2247 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1377780316085/jetty3/index
   [junit4]   2> 780524 T2247 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=646070326, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 780524 T2247 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67c0c5ff lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476aab8f),segFN=segments_1,generation=1}
   [junit4]   2> 780525 T2247 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 780527 T2247 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 780528 T2247 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 780528 T2247 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 780528 T2247 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 780529 T2247 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 780529 T2247 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 780529 T2247 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 780530 T2247 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 780530 T2247 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 780531 T2247 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 780531 T2247 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 780532 T2247 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 780532 T2247 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 780533 T2247 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 780533 T2247 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 780534 T2247 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 780537 T2247 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 780537 T2247 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 780538 T2247 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=839210927, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 780539 T2247 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67c0c5ff lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476aab8f),segFN=segments_1,generation=1}
   [junit4]   2> 780539 T2247 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 780539 T2247 oass.SolrIndexSearcher.<init> Opening Searcher@8e039420 main
   [junit4]   2> 780543 T2248 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8e039420 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 780544 T2247 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 780544 T2247 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53973 collection:collection1 shard:shard1
   [junit4]   2> 780545 T2247 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 780551 T2247 oasc.ZkController.register We are http://127.0.0.1:53973/collection1/ and leader is http://127.0.0.1:45466/collection1/
   [junit4]   2> 780551 T2247 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53973
   [junit4]   2> 780551 T2247 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 780552 T2247 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C797 name=collection1 org.apache.solr.core.SolrCore@e424de17 url=http://127.0.0.1:53973/collection1 node=127.0.0.1:53973_ C797_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:53973, state=down, node_name=127.0.0.1:53973_, core=collection1}
   [junit4]   2> 780552 T2249 C797 P53973 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 780552 T2247 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 780553 T2249 C797 P53973 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 780553 T2249 C797 P53973 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 780554 T2172 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 780554 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 780554 T2249 C797 P53973 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 780555 T2172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 780557 T2249 C797 P53973 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 780562 T2208 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 780626 T2172 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 780628 T2172 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42511
   [junit4]   2> 780629 T2172 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 780629 T2172 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 780630 T2172 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1377780328561
   [junit4]   2> 780630 T2172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1377780328561/'
   [junit4]   2> 780638 T2172 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1377780328561/solr.xml
   [junit4]   2> 780669 T2172 oasc.CoreContainer.<init> New CoreContainer -386938147
   [junit4]   2> 780669

[...truncated too long message...]

  {
   [junit4]   1>            "base_url":"http://127.0.0.1:45466",
   [junit4]   1>            "node_name":"127.0.0.1:45466_",
   [junit4]   1>            "core":"onenodecollectioncore"}
   [junit4]   1>   /solr/aliases.json (0)
   [junit4]   1>   /solr/live_nodes (5)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:45466_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:35466_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:39818_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:42511_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:53973_ (0)
   [junit4]   1>   /solr/clusterstate.json (0)
   [junit4]   1>   DATA:
   [junit4]   1>       {
   [junit4]   1>         "control_collection":{
   [junit4]   1>           "shards":{"shard1":{
   [junit4]   1>               "range":"80000000-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:39818",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:39818_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "onenodecollection":{
   [junit4]   1>           "shards":{"shard1":{
   [junit4]   1>               "range":"80000000-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:45466",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:45466_",
   [junit4]   1>                   "roles":"none",
   [junit4]   1>                   "core":"onenodecollectioncore",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "collection1":{
   [junit4]   1>           "shards":{
   [junit4]   1>             "shard1":{
   [junit4]   1>               "range":"80000000-ffffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node1":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:45466",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:45466_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node3":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:53973",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:53973_",
   [junit4]   1>                   "core":"collection1"}}},
   [junit4]   1>             "shard2":{
   [junit4]   1>               "range":"0-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node2":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:35466",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:35466_",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node4":{
   [junit4]   1>                   "base_url":"http://127.0.0.1:42511",
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "node_name":"127.0.0.1:42511_",
   [junit4]   1>                   "core":"collection1"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"}}
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=8B26208A68D23A8C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PE -Dtests.timezone=Europe/Simferopol -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   28.3s J1 | BasicDistributedZk2Test.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:35466/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8B26208A68D23A8C:AC0AE921F8D5AB0]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 796364 T2172 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 28290 T2171 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 797123 T2263 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 797123 T2263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 797123 T2263 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_PE, timezone=Europe/Simferopol
   [junit4]   2> NOTE: Linux 3.2.0-52-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=3,free=52776696,total=187695104
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsBasicDistributedZk2Test, DirectSolrSpellCheckerTest, AddBlockUpdateTest, DefaultValueUpdateProcessorTest, IndexSchemaRuntimeFieldTest, TestSolrCoreProperties, TestDynamicFieldCollectionResource, PathHierarchyTokenizerFactoryTest, TestRandomMergePolicy, UniqFieldsUpdateProcessorFactoryTest, TestFiltering, SignatureUpdateProcessorFactoryTest, SolrPluginUtilsTest, BasicZkTest, TestQueryTypes, CollectionsAPIDistributedZkTest, ZkNodePropsTest, OutputWriterTest, TestBinaryResponseWriter, HardAutoCommitTest, TestFoldingMultitermQuery, TestZkChroot, TestSolrXml, TestQuerySenderListener, SolrTestCaseJ4Test, TestRandomFaceting, HighlighterConfigTest, SolrCoreCheckLockOnStartupTest, TestXIncludeConfig, TestSolrDeletionPolicy2, DirectUpdateHandlerTest, FullSolrCloudDistribCmdsTest, TestFastWriter, TestCloudManagedSchemaAddField, TestSweetSpotSimilarityFactory, PreAnalyzedUpdateProcessorTest, SpellingQueryConverterTest, TestIndexingPerformance, PrimUtilsTest, TestMultiCoreConfBootstrap, DeleteShardTest, PluginInfoTest, TestPartialUpdateDeduplication, SolrCmdDistributorTest, LeaderElectionIntegrationTest, TestDFRSimilarityFactory, TestRecoveryHdfs, TestJmxIntegration, DocValuesMultiTest, TestReload, DocumentAnalysisRequestHandlerTest, ConvertedLegacyTest, TestOmitPositions, TestSolr4Spatial, FileUtilsTest, DisMaxRequestHandlerTest, TestRangeQuery, TestFastLRUCache, NotRequiredUniqueKeyTest, TestSolrJ, TestDefaultSearchFieldResource, TestLazyCores, TestStressRecovery, SpellCheckComponentTest, OpenExchangeRatesOrgProviderTest, AddSchemaFieldsUpdateProcessorFactoryTest, AlternateDirectoryTest, CurrencyFieldOpenExchangeTest, TestSchemaResource, TestRTGBase, BadIndexSchemaTest, TestFuzzyAnalyzedSuggestions, XmlUpdateRequestHandlerTest, ZkSolrClientTest, TestNumberUtils, TestQuerySenderNoQuery, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test]
   [junit4] Completed on J1 in 29.31s, 1 test, 1 error <<< FAILURES!

[...truncated 604 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:382: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 324 suites, 1408 tests, 1 error, 35 ignored (6 assumptions)

Total time: 43 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure