You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/10/20 16:49:08 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 7966 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/7966/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

1 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Expected content type application/octet-stream but got text/html;charset=ISO-8859-1. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 Can not find: /onenodecollectioncore/update</title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /onenodecollectioncore/update. Reason: <pre>    Can not find: /onenodecollectioncore/update</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected content type application/octet-stream but got text/html;charset=ISO-8859-1. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 Can not find: /onenodecollectioncore/update</title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /onenodecollectioncore/update. Reason:
<pre>    Can not find: /onenodecollectioncore/update</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([734040E0B35ECC01:F2A6CEF8C401AC3D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)




Build Log:
[...truncated 10181 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 707563 T1876 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 707578 T1876 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1382279878072
   [junit4]   2> 707580 T1876 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 707581 T1877 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 707681 T1876 oasc.ZkTestServer.run start zk server on port:56415
   [junit4]   2> 707682 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707730 T1883 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a0d5d34d name:ZooKeeperConnection Watcher:127.0.0.1:56415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707730 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707731 T1876 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 707747 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707763 T1885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fd44b03 name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707763 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707764 T1876 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 707767 T1876 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 707769 T1876 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 707773 T1876 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 707777 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 707778 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 707783 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 707784 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 707788 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 707789 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 707793 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 707794 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 707798 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 707799 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 707804 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 707804 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 707812 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 707818 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 707839 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 707840 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 707847 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 707848 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 707853 T1876 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 707854 T1876 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 708025 T1876 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 708045 T1876 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49887
   [junit4]   2> 708047 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 708047 T1876 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 708048 T1876 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357
   [junit4]   2> 708049 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/'
   [junit4]   2> 708062 T1876 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/solr.xml
   [junit4]   2> 708107 T1876 oasc.CoreContainer.<init> New CoreContainer -141396878
   [junit4]   2> 708108 T1876 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/]
   [junit4]   2> 708110 T1876 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 708111 T1876 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 708112 T1876 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 708113 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 708114 T1876 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 708115 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 708116 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 708116 T1876 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 708117 T1876 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 708120 T1876 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 708121 T1876 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 708124 T1876 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56415/solr
   [junit4]   2> 708125 T1876 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 708128 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 708147 T1896 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d0a50af9 name:ZooKeeperConnection Watcher:127.0.0.1:56415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 708148 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 708151 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 708154 T1898 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e4a9ba3 name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 708155 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 708160 T1876 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 708169 T1876 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 708173 T1876 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 708178 T1876 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49887_
   [junit4]   2> 708180 T1876 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49887_
   [junit4]   2> 708187 T1876 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 708195 T1876 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 708200 T1876 oasc.Overseer.start Overseer (id=90589094089588739-127.0.0.1:49887_-n_0000000000) starting
   [junit4]   2> 708206 T1876 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 708212 T1900 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 708214 T1876 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 708218 T1876 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 708220 T1876 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 708231 T1899 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 708241 T1901 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 708241 T1901 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 708249 T1901 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 709743 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 709744 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49887",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49887_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 709744 T1899 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 709745 T1899 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 709750 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 710250 T1901 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 710251 T1901 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/collection1
   [junit4]   2> 710251 T1901 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 710252 T1901 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 710253 T1901 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 710258 T1901 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/collection1/'
   [junit4]   2> 710259 T1901 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/collection1/lib/classes/' to classloader
   [junit4]   2> 710259 T1901 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/collection1/lib/README' to classloader
   [junit4]   2> 710332 T1901 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 710397 T1901 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 710398 T1901 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 710409 T1901 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 710968 T1901 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 710988 T1901 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 710993 T1901 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711039 T1901 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711049 T1901 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711066 T1901 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711069 T1901 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711070 T1901 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711072 T1901 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 711076 T1901 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 711077 T1901 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 711079 T1901 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 711081 T1901 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1382279878357/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/control/data/
   [junit4]   2> 711082 T1901 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fa6d793
   [junit4]   2> 711083 T1901 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/control/data
   [junit4]   2> 711084 T1901 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/control/data/index/
   [junit4]   2> 711084 T1901 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 711085 T1901 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/control/data/index
   [junit4]   2> 711091 T1901 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 711097 T1901 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b574f7be lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50153fe3),segFN=segments_1,generation=1}
   [junit4]   2> 711097 T1901 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711109 T1901 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 711110 T1901 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 711110 T1901 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 711111 T1901 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711113 T1901 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711116 T1901 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 711117 T1901 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711118 T1901 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711119 T1901 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 711124 T1901 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 711125 T1901 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 711126 T1901 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 711126 T1901 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 711127 T1901 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 711128 T1901 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 711129 T1901 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 711135 T1901 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 711135 T1901 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 711136 T1901 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 711137 T1901 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@b574f7be lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@50153fe3),segFN=segments_1,generation=1}
   [junit4]   2> 711137 T1901 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711138 T1901 oass.SolrIndexSearcher.<init> Opening Searcher@3910e4fb main
   [junit4]   2> 711140 T1902 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3910e4fb main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 711142 T1901 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 711143 T1901 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49887 collection:control_collection shard:shard1
   [junit4]   2> 711151 T1901 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 711164 T1901 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 711167 T1901 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 711168 T1901 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 711168 T1901 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49887/collection1/
   [junit4]   2> 711170 T1901 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 711170 T1901 oasc.SyncStrategy.syncToMe http://127.0.0.1:49887/collection1/ has no replicas
   [junit4]   2> 711171 T1901 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49887/collection1/ shard1
   [junit4]   2> 711171 T1901 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 711256 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711263 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 711285 T1901 oasc.ZkController.register We are http://127.0.0.1:49887/collection1/ and leader is http://127.0.0.1:49887/collection1/
   [junit4]   2> 711286 T1901 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49887
   [junit4]   2> 711287 T1901 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 711289 T1901 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 711289 T1901 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 711292 T1901 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711293 T1876 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 711294 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 711296 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 711303 T1905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf1c1c9c name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 711304 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 711307 T1876 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 711313 T1876 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 711420 T1876 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 711422 T1876 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56144
   [junit4]   2> 711423 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 711424 T1876 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 711424 T1876 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806
   [junit4]   2> 711425 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/'
   [junit4]   2> 711433 T1876 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/solr.xml
   [junit4]   2> 711474 T1876 oasc.CoreContainer.<init> New CoreContainer -931239049
   [junit4]   2> 711474 T1876 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/]
   [junit4]   2> 711477 T1876 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 711496 T1876 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 711497 T1876 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 711498 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 711499 T1876 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 711500 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 711501 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 711502 T1876 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 711503 T1876 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 711505 T1876 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 711506 T1876 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 711509 T1876 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56415/solr
   [junit4]   2> 711510 T1876 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 711513 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 711521 T1916 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ec7093d3 name:ZooKeeperConnection Watcher:127.0.0.1:56415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 711522 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 711527 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 711531 T1918 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eb6a244 name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 711532 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 711544 T1876 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 712552 T1876 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56144_
   [junit4]   2> 712555 T1876 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56144_
   [junit4]   2> 712559 T1898 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 712559 T1918 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 712559 T1905 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 712566 T1919 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 712567 T1919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 712569 T1919 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 712769 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 712770 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49887",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49887_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 712774 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56144",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56144_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 712775 T1899 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 712775 T1899 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 712779 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 712780 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 712782 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 713570 T1919 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 713571 T1919 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/collection1
   [junit4]   2> 713571 T1919 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 713572 T1919 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 713573 T1919 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 713574 T1919 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/collection1/'
   [junit4]   2> 713575 T1919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/collection1/lib/classes/' to classloader
   [junit4]   2> 713575 T1919 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/collection1/lib/README' to classloader
   [junit4]   2> 713633 T1919 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 713698 T1919 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 713701 T1919 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 713723 T1919 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 714221 T1919 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 714229 T1919 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 714232 T1919 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 714261 T1919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 714266 T1919 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 714271 T1919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 714275 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 714278 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 714281 T1919 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 714289 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 714290 T1919 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 714290 T1919 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 714290 T1919 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1382279881806/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty1/
   [junit4]   2> 714291 T1919 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fa6d793
   [junit4]   2> 714292 T1919 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty1
   [junit4]   2> 714292 T1919 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty1/index/
   [junit4]   2> 714293 T1919 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 714304 T1919 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty1/index
   [junit4]   2> 714304 T1919 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 714312 T1919 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@f21b0c0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cd0dd4b9),segFN=segments_1,generation=1}
   [junit4]   2> 714312 T1919 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 714319 T1919 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 714321 T1919 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 714322 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 714322 T1919 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 714323 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 714324 T1919 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 714325 T1919 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 714326 T1919 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 714327 T1919 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 714328 T1919 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 714329 T1919 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 714329 T1919 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 714330 T1919 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 714335 T1919 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 714336 T1919 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 714338 T1919 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 714342 T1919 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 714342 T1919 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 714343 T1919 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 714345 T1919 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@f21b0c0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cd0dd4b9),segFN=segments_1,generation=1}
   [junit4]   2> 714346 T1919 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 714346 T1919 oass.SolrIndexSearcher.<init> Opening Searcher@9627a944 main
   [junit4]   2> 714351 T1919 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 714351 T1919 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56144 collection:collection1 shard:shard1
   [junit4]   2> 714353 T1920 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9627a944 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 714356 T1919 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 714366 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 714370 T1919 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 714371 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 714372 T1919 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56144/collection1/
   [junit4]   2> 714372 T1919 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 714373 T1919 oasc.SyncStrategy.syncToMe http://127.0.0.1:56144/collection1/ has no replicas
   [junit4]   2> 714373 T1919 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56144/collection1/ shard1
   [junit4]   2> 714374 T1919 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 715794 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 715800 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 715800 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 715800 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 715806 T1919 oasc.ZkController.register We are http://127.0.0.1:56144/collection1/ and leader is http://127.0.0.1:56144/collection1/
   [junit4]   2> 715806 T1919 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56144
   [junit4]   2> 715807 T1919 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 715807 T1919 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 715808 T1919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 715810 T1919 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 715811 T1876 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 715812 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 715893 T1876 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 715896 T1876 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54950
   [junit4]   2> 715897 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 715898 T1876 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 715898 T1876 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306
   [junit4]   2> 715899 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/'
   [junit4]   2> 715908 T1876 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/solr.xml
   [junit4]   2> 715971 T1876 oasc.CoreContainer.<init> New CoreContainer -2091626483
   [junit4]   2> 715972 T1876 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/]
   [junit4]   2> 715974 T1876 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 715975 T1876 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 715975 T1876 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 715976 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 715977 T1876 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 715978 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 715979 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 715980 T1876 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 715981 T1876 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 715983 T1876 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 715984 T1876 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 715987 T1876 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56415/solr
   [junit4]   2> 715988 T1876 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 715991 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 715993 T1932 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd9d6b2 name:ZooKeeperConnection Watcher:127.0.0.1:56415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 715995 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 716000 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 716019 T1934 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33aee3e1 name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 716021 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 716025 T1876 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 717030 T1876 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54950_
   [junit4]   2> 717032 T1876 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54950_
   [junit4]   2> 717036 T1905 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 717036 T1934 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 717037 T1898 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 717040 T1918 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 717044 T1935 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 717044 T1935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 717045 T1935 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 717305 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 717306 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56144",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:56144_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 717311 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54950",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54950_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 717312 T1899 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 717312 T1899 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 717316 T1934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 717316 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 717316 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 717316 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 718046 T1935 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 718046 T1935 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/collection1
   [junit4]   2> 718047 T1935 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 718047 T1935 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 718048 T1935 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 718049 T1935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/collection1/'
   [junit4]   2> 718050 T1935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/collection1/lib/classes/' to classloader
   [junit4]   2> 718050 T1935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/collection1/lib/README' to classloader
   [junit4]   2> 718099 T1935 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 718151 T1935 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 718153 T1935 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 718161 T1935 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 718658 T1935 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 718667 T1935 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 718671 T1935 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 718710 T1935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 718731 T1935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 718742 T1935 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 718748 T1935 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 718749 T1935 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 718751 T1935 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 718755 T1935 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 718755 T1935 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 718756 T1935 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 718756 T1935 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1382279886306/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty2/
   [junit4]   2> 718757 T1935 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fa6d793
   [junit4]   2> 718758 T1935 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty2
   [junit4]   2> 718758 T1935 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty2/index/
   [junit4]   2> 718759 T1935 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 718760 T1935 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty2/index
   [junit4]   2> 718760 T1935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 718761 T1935 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4f5bc0c5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de87e79d),segFN=segments_1,generation=1}
   [junit4]   2> 718761 T1935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 718765 T1935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 718766 T1935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 718766 T1935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 718767 T1935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 718767 T1935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 718767 T1935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 718768 T1935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 718768 T1935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 718769 T1935 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 718769 T1935 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 718770 T1935 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 718770 T1935 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 718771 T1935 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 718771 T1935 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 718772 T1935 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 718774 T1935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 718780 T1935 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 718780 T1935 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 718781 T1935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 718781 T1935 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4f5bc0c5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@de87e79d),segFN=segments_1,generation=1}
   [junit4]   2> 718782 T1935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 718783 T1935 oass.SolrIndexSearcher.<init> Opening Searcher@8031c114 main
   [junit4]   2> 718787 T1936 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8031c114 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 718788 T1935 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 718789 T1935 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54950 collection:collection1 shard:shard2
   [junit4]   2> 718790 T1935 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 718814 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 718821 T1935 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 718822 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 718823 T1935 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54950/collection1/
   [junit4]   2> 718823 T1935 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 718824 T1935 oasc.SyncStrategy.syncToMe http://127.0.0.1:54950/collection1/ has no replicas
   [junit4]   2> 718824 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54950/collection1/ shard2
   [junit4]   2> 718824 T1935 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 718827 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 718834 T1934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 718835 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 718835 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 718835 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 720343 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 720347 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 720347 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 720347 T1934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 720347 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 720394 T1935 oasc.ZkController.register We are http://127.0.0.1:54950/collection1/ and leader is http://127.0.0.1:54950/collection1/
   [junit4]   2> 720395 T1935 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54950
   [junit4]   2> 720395 T1935 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 720395 T1935 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 720396 T1935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 720399 T1935 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 720403 T1876 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 720404 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 720567 T1876 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 720570 T1876 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40243
   [junit4]   2> 720571 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 720572 T1876 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 720572 T1876 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899
   [junit4]   2> 720573 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/'
   [junit4]   2> 720583 T1876 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/solr.xml
   [junit4]   2> 720631 T1876 oasc.CoreContainer.<init> New CoreContainer -472118491
   [junit4]   2> 720632 T1876 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/]
   [junit4]   2> 720634 T1876 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 720635 T1876 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 720636 T1876 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 720636 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 720637 T1876 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 720638 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 720639 T1876 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 720640 T1876 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 720641 T1876 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 720642 T1876 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 720643 T1876 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 720644 T1876 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56415/solr
   [junit4]   2> 720645 T1876 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 720646 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 720647 T1948 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90748747 name:ZooKeeperConnection Watcher:127.0.0.1:56415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 720648 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 720650 T1876 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 720653 T1950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ccf55d9e name:ZooKeeperConnection Watcher:127.0.0.1:56415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 720653 T1876 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 720659 T1876 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 721663 T1876 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40243_
   [junit4]   2> 721664 T1876 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40243_
   [junit4]   2> 721667 T1905 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 721668 T1918 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 721668 T1934 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 721668 T1898 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 721668 T1950 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 721676 T1951 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 721678 T1951 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 721681 T1951 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 721863 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 721864 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54950",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54950_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 721868 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40243",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40243_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node3"}
   [junit4]   2> 721869 T1899 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 721869 T1899 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 721876 T1934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 721876 T1918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 721876 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 721876 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 721876 T1898 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 722682 T1951 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 722682 T1951 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/collection1
   [junit4]   2> 722683 T1951 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 722684 T1951 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 722684 T1951 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 722686 T1951 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/collection1/'
   [junit4]   2> 722687 T1951 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/collection1/lib/classes/' to classloader
   [junit4]   2> 722687 T1951 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/collection1/lib/README' to classloader
   [junit4]   2> 722730 T1951 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 722772 T1951 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 722773 T1951 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 722784 T1951 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 723181 T1951 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 723186 T1951 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 723189 T1951 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 723203 T1951 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 723207 T1951 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 723212 T1951 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 723213 T1951 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 723213 T1951 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 723214 T1951 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 723215 T1951 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 723215 T1951 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 723216 T1951 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 723216 T1951 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1382279890899/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty3/
   [junit4]   2> 723217 T1951 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fa6d793
   [junit4]   2> 723218 T1951 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty3
   [junit4]   2> 723218 T1951 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty3/index/
   [junit4]   2> 723218 T1951 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 723219 T1951 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1382279878071/jetty3/index
   [junit4]   2> 723219 T1951 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 723220 T1951 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@119f6e73 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1226c0c9),segFN=segments_1,generation=1}
   [junit4]   2> 723221 T1951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 723223 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 723224 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 723224 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 723225 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 723225 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 723226 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 723226 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 723227 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 723227 T1951 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 723228 T1951 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 723228 T1951 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 723229 T1951 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 723229 T1951 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 723230 T1951 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 723230 T1951 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 723232 T1951 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 723235 T1951 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 723235 T1951 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 723236 T1951 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=15, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 723237 T1951 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@119f6e73 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1226c0c9),segFN=segments_1,generation=1}
   [junit4]   2> 723237 T1951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 723238 T1951 oass.SolrIndexSearcher.<init> Opening Searcher@89f6e5cf main
   [junit4]   2> 723241 T1952 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@89f6e5cf main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 723242 T1951 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 723243 T1951 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40243 collection:collection1 shard:shard1
   [junit4]   2> 723247 T1951 oasc.ZkController.register We are http://127.0.0.1:40243/collection1/ and leader is http://127.0.0.1:56144/collection1/
   [junit4]   2> 723247 T1951 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40243
   [junit4]   2> 723247 T1951 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 723248 T1951 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C1297 name=collection1 org.apache.solr.core.SolrCore@e885d799 url=http://127.0.0.1:40243/collection1 node=127.0.0.1:40243_ C1297_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40243, core=collection1, node_name=127.0.0.1:40243_}
   [junit4]   2> 723249 T1953 C1297 P40243 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 723249 T1951 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 723249 T1953 C1297 P40243 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 723250 T1953 C1297 P40243 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 723250 T1876 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 723251 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 723250 T1953 C1297 P40243 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 723255 T1912 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 723370 T1876 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 723379 T1876 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40321
   [junit4]   2> 723383 T1876 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 723384 T1876 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 723385 T1876 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1382279893746
   [junit4]   2> 723386 T1876 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1382279893746/'
   [junit4]   2> 723392 T1899 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 723393 T1899 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40243",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40243_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node3"}
   [junit4]   2> 723400 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 723400 T1934 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)

[...truncated too long message...]


   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:56144",
   [junit4]   1>                   "core":"onenodecollectioncore",
   [junit4]   1>                   "roles":"none",
   [junit4]   1>                   "node_name":"127.0.0.1:56144_",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":{"name":"compositeId"},
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "collection1":{
   [junit4]   1>           "shards":{
   [junit4]   1>             "shard1":{
   [junit4]   1>               "range":"80000000-ffffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node1":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:56144",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:56144_",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node3":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:40243",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:40243_"}}},
   [junit4]   1>             "shard2":{
   [junit4]   1>               "range":"0-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{
   [junit4]   1>                 "core_node2":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:54950",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:54950_",
   [junit4]   1>                   "leader":"true"},
   [junit4]   1>                 "core_node4":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:40321",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:40321_"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":{"name":"compositeId"},
   [junit4]   1>           "replicationFactor":"1"}}
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=734040E0B35ECC01 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_AR -Dtests.timezone=Pacific/Honolulu -Dtests.file.encoding=UTF-8
   [junit4] ERROR   32.9s J1 | BasicDistributedZk2Test.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected content type application/octet-stream but got text/html;charset=ISO-8859-1. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 404 Can not find: /onenodecollectioncore/update</title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 404</h2>
   [junit4]    > <p>Problem accessing /onenodecollectioncore/update. Reason:
   [junit4]    > <pre>    Can not find: /onenodecollectioncore/update</pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([734040E0B35ECC01:F2A6CEF8C401AC3D]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)
   [junit4]   2> 740501 T1876 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 32945 T1875 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> 740526 T1950 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 741622 T1967 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=es_AR, timezone=Pacific/Honolulu
   [junit4]   2> NOTE: Linux 3.8.0-31-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=2,free=9289984,total=232521728
   [junit4]   2> NOTE: All tests run in this JVM: [HardAutoCommitTest, DateFieldTest, InfoHandlerTest, DebugComponentTest, TestSolrCoreProperties, SpellCheckComponentTest, StatsComponentTest, TestRandomFaceting, TestStressLucene, TestIBSimilarityFactory, TestZkChroot, MultiTermTest, PathHierarchyTokenizerFactoryTest, TestFoldingMultitermQuery, TestRangeQuery, SolrCoreCheckLockOnStartupTest, ConvertedLegacyTest, PluginInfoTest, ChaosMonkeySafeLeaderTest, TestDistributedSearch, DirectSolrConnectionTest, TestDynamicFieldCollectionResource, TestArbitraryIndexDir, ZkSolrClientTest, TestNonNRTOpen, BadIndexSchemaTest, TestIndexSearcher, ZkNodePropsTest, PingRequestHandlerTest, OpenExchangeRatesOrgProviderTest, HdfsBasicDistributedZk2Test, TestUpdate, TestFastOutputStream, TestRecovery, HdfsChaosMonkeySafeLeaderTest, ShardRoutingCustomTest, SignatureUpdateProcessorFactoryTest, DocumentAnalysisRequestHandlerTest, LeaderElectionIntegrationTest, CachingDirectoryFactoryTest, XmlUpdateRequestHandlerTest, BasicZkTest, SolrRequestParserTest, TestAddFieldRealTimeGet, AddBlockUpdateTest, SolrTestCaseJ4Test, RequiredFieldsTest, TestFastWriter, TestSolr4Spatial, SimplePostToolTest, SuggesterTSTTest, NotRequiredUniqueKeyTest, HighlighterTest, TestFiltering, XsltUpdateRequestHandlerTest, RAMDirectoryFactoryTest, NoCacheHeaderTest, HighlighterConfigTest, SuggesterWFSTTest, TestFieldResource, OutputWriterTest, SpellCheckCollatorTest, CurrencyFieldOpenExchangeTest, TestDFRSimilarityFactory, TestNumberUtils, SpellingQueryConverterTest, CustomCollectionTest, CoreContainerCoreInitFailuresTest, PrimUtilsTest, CoreMergeIndexesAdminHandlerTest, TestRTGBase, TestBinaryField, TestPhraseSuggestions, TestSolrJ, TestCollationField, TestUtils, TestAtomicUpdateErrorCases, TestCodecSupport, HdfsLockFactoryTest, NumericFieldsTest, AddSchemaFieldsUpdateProcessorFactoryTest, BasicDistributedZk2Test]
   [junit4] Completed on J1 in 34.26s, 1 test, 1 error <<< FAILURES!

[...truncated 572 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:422: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:415: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:483: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 326 suites, 1449 tests, 1 error, 35 ignored (6 assumptions)

Total time: 48 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure