You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/08/18 18:16:34 UTC

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_25) - Build # 6984 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6984/
Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:57172/dboil/b/collection1lastClient and got 5 from http://127.0.0.1:41835/dboil/b/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:57172/dboil/b/collection1lastClient and got 5 from http://127.0.0.1:41835/dboil/b/collection1
	at __randomizedtesting.SeedInfo.seed([F9893A201817B35E:786FB4386F48D362]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1128)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9686 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> 254446 T1190 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /dboil/b
   [junit4]   2> 254450 T1190 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1376841111595
   [junit4]   2> 254451 T1190 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 254451 T1191 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 254551 T1190 oasc.ZkTestServer.run start zk server on port:38962
   [junit4]   2> 254552 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 254628 T1197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a856dc name:ZooKeeperConnection Watcher:127.0.0.1:38962 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 254628 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 254628 T1190 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 254633 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 254633 T1199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d5070d name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 254634 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 254634 T1190 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 254636 T1190 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 254638 T1190 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 254639 T1190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 254641 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 254641 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 254644 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 254644 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 254647 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 254647 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 254649 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 254649 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 254651 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 254651 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 254653 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 254653 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 254655 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 254655 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 254660 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 254660 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 254662 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 254662 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 254664 T1190 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 254664 T1190 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 254730 T1190 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 254733 T1190 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33512
   [junit4]   2> 254734 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 254734 T1190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 254734 T1190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810
   [junit4]   2> 254735 T1190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/'
   [junit4]   2> 254745 T1190 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/solr.xml
   [junit4]   2> 254785 T1190 oasc.CoreContainer.<init> New CoreContainer 30897530
   [junit4]   2> 254786 T1190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/]
   [junit4]   2> 254787 T1190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 254787 T1190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 254787 T1190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 254788 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 254788 T1190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 254788 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 254788 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 254789 T1190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 254789 T1190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 254789 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 254792 T1190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38962/solr
   [junit4]   2> 254792 T1190 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 254793 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 254794 T1210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bee95 name:ZooKeeperConnection Watcher:127.0.0.1:38962 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 254795 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 254796 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 254798 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 254799 T1212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59eaef name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 254799 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 254800 T1190 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 254802 T1190 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 254804 T1190 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 254805 T1190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33512_dboil%2Fb
   [junit4]   2> 254806 T1190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33512_dboil%2Fb
   [junit4]   2> 254808 T1190 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 254811 T1190 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 254812 T1190 oasc.Overseer.start Overseer (id=90232659089620995-127.0.0.1:33512_dboil%2Fb-n_0000000000) starting
   [junit4]   2> 254814 T1190 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 254816 T1214 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 254817 T1190 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 254818 T1190 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 254819 T1190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 254821 T1213 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 254823 T1215 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 254823 T1215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 254824 T1215 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 256324 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 256325 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33512_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33512/dboil/b"}
   [junit4]   2> 256325 T1213 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 256325 T1213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 256329 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 256825 T1215 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 256825 T1215 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/collection1
   [junit4]   2> 256826 T1215 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 256827 T1215 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 256828 T1215 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 256830 T1215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/collection1/'
   [junit4]   2> 256831 T1215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/collection1/lib/README' to classloader
   [junit4]   2> 256832 T1215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/collection1/lib/classes/' to classloader
   [junit4]   2> 256884 T1215 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 256918 T1215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 256919 T1215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 256924 T1215 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 257276 T1215 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 257276 T1215 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 257277 T1215 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 257282 T1215 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 257284 T1215 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 257293 T1215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 257296 T1215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 257298 T1215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 257299 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 257299 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 257299 T1215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 257300 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 257300 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 257301 T1215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 257301 T1215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.SyncSliceTest-controljetty-1376841111810/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1376841111594/control/data/
   [junit4]   2> 257301 T1215 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092c6
   [junit4]   2> 257301 T1215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/control/data
   [junit4]   2> 257302 T1215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1376841111594/control/data/index/
   [junit4]   2> 257302 T1215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1376841111594/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 257302 T1215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/control/data/index
   [junit4]   2> 257302 T1215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 257303 T1215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@dc7bb0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1a39c8d),segFN=segments_1,generation=1}
   [junit4]   2> 257303 T1215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 257305 T1215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 257305 T1215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 257306 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 257306 T1215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 257306 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 257306 T1215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 257306 T1215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 257307 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 257307 T1215 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 257307 T1215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 257307 T1215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 257308 T1215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 257308 T1215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 257308 T1215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 257308 T1215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 257311 T1215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 257313 T1215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 257313 T1215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 257313 T1215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@e7553b
   [junit4]   2> 257314 T1215 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@dc7bb0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1a39c8d),segFN=segments_1,generation=1}
   [junit4]   2> 257314 T1215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 257314 T1215 oass.SolrIndexSearcher.<init> Opening Searcher@17c3d70 main
   [junit4]   2> 257316 T1216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17c3d70 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 257316 T1215 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 257317 T1215 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33512/dboil/b collection:control_collection shard:shard1
   [junit4]   2> 257317 T1215 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 257323 T1215 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 257328 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 257330 T1215 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 257330 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 257330 T1215 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33512/dboil/b/collection1/
   [junit4]   2> 257331 T1215 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 257331 T1215 oasc.SyncStrategy.syncToMe http://127.0.0.1:33512/dboil/b/collection1/ has no replicas
   [junit4]   2> 257331 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33512/dboil/b/collection1/ shard1
   [junit4]   2> 257331 T1215 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 257833 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 257839 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 257887 T1215 oasc.ZkController.register We are http://127.0.0.1:33512/dboil/b/collection1/ and leader is http://127.0.0.1:33512/dboil/b/collection1/
   [junit4]   2> 257887 T1215 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33512/dboil/b
   [junit4]   2> 257887 T1215 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 257887 T1215 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 257887 T1215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 257889 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 257890 T1190 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 257890 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 257891 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 257893 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 257894 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 257896 T1219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ace24c name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 257897 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 257897 T1190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 257899 T1190 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 258007 T1190 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 258009 T1190 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33184
   [junit4]   2> 258009 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 258010 T1190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 258010 T1190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043
   [junit4]   2> 258010 T1190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/'
   [junit4]   2> 258020 T1190 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/solr.xml
   [junit4]   2> 258055 T1190 oasc.CoreContainer.<init> New CoreContainer 32730637
   [junit4]   2> 258055 T1190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/]
   [junit4]   2> 258056 T1190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 258056 T1190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 258056 T1190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 258057 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 258057 T1190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 258057 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 258057 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 258058 T1190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 258058 T1190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 258058 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 258061 T1190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38962/solr
   [junit4]   2> 258061 T1190 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 258062 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 258063 T1230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d523bd name:ZooKeeperConnection Watcher:127.0.0.1:38962 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 258063 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 258064 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 258067 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 258068 T1232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@154108d name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 258068 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 258070 T1190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 259072 T1190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33184_dboil%2Fb
   [junit4]   2> 259074 T1190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33184_dboil%2Fb
   [junit4]   2> 259079 T1219 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 259079 T1212 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 259079 T1232 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 259079 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 259086 T1233 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 259087 T1233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 259088 T1233 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 259343 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 259344 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33512_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33512/dboil/b"}
   [junit4]   2> 259346 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33184_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33184/dboil/b"}
   [junit4]   2> 259346 T1213 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 259347 T1213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 259349 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 259349 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 259349 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 260088 T1233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 260089 T1233 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/collection1
   [junit4]   2> 260089 T1233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 260089 T1233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 260089 T1233 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 260090 T1233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/collection1/'
   [junit4]   2> 260091 T1233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/collection1/lib/README' to classloader
   [junit4]   2> 260091 T1233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/collection1/lib/classes/' to classloader
   [junit4]   2> 260125 T1233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 260163 T1233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 260165 T1233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 260170 T1233 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 260533 T1233 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 260533 T1233 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 260534 T1233 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 260543 T1233 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 260546 T1233 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 260556 T1233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 260559 T1233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 260562 T1233 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 260563 T1233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 260563 T1233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 260563 T1233 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 260564 T1233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 260564 T1233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 260564 T1233 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 260565 T1233 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1376841115043/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty1/
   [junit4]   2> 260565 T1233 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092c6
   [junit4]   2> 260565 T1233 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty1
   [junit4]   2> 260565 T1233 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty1/index/
   [junit4]   2> 260566 T1233 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 260566 T1233 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty1/index
   [junit4]   2> 260566 T1233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 260567 T1233 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a63a80 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11cce11),segFN=segments_1,generation=1}
   [junit4]   2> 260567 T1233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 260569 T1233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 260570 T1233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 260570 T1233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 260570 T1233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 260570 T1233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 260570 T1233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 260571 T1233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 260571 T1233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 260571 T1233 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 260572 T1233 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 260572 T1233 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 260572 T1233 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 260572 T1233 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 260573 T1233 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 260573 T1233 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 260575 T1233 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 260577 T1233 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 260578 T1233 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 260578 T1233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@dbeedc
   [junit4]   2> 260579 T1233 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a63a80 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11cce11),segFN=segments_1,generation=1}
   [junit4]   2> 260579 T1233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 260579 T1233 oass.SolrIndexSearcher.<init> Opening Searcher@3a2586 main
   [junit4]   2> 260582 T1234 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a2586 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 260583 T1233 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 260584 T1233 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33184/dboil/b collection:collection1 shard:shard1
   [junit4]   2> 260584 T1233 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 260586 T1233 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 260677 T1233 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 260681 T1233 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 260681 T1233 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 260681 T1233 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33184/dboil/b/collection1/
   [junit4]   2> 260682 T1233 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 260682 T1233 oasc.SyncStrategy.syncToMe http://127.0.0.1:33184/dboil/b/collection1/ has no replicas
   [junit4]   2> 260682 T1233 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33184/dboil/b/collection1/ shard1
   [junit4]   2> 260683 T1233 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 260854 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 260865 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 260865 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 260865 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 260891 T1233 oasc.ZkController.register We are http://127.0.0.1:33184/dboil/b/collection1/ and leader is http://127.0.0.1:33184/dboil/b/collection1/
   [junit4]   2> 260891 T1233 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33184/dboil/b
   [junit4]   2> 260891 T1233 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 260892 T1233 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 260892 T1233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 260894 T1233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 260896 T1190 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 260896 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 260897 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 260978 T1190 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 260980 T1190 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48187
   [junit4]   2> 260981 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 260981 T1190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 260981 T1190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046
   [junit4]   2> 260982 T1190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/'
   [junit4]   2> 260995 T1190 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/solr.xml
   [junit4]   2> 261031 T1190 oasc.CoreContainer.<init> New CoreContainer 22030877
   [junit4]   2> 261032 T1190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/]
   [junit4]   2> 261032 T1190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 261033 T1190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 261033 T1190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 261033 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 261033 T1190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 261034 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 261034 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 261034 T1190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 261034 T1190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 261035 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 261037 T1190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38962/solr
   [junit4]   2> 261038 T1190 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 261038 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 261040 T1246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@868a0d name:ZooKeeperConnection Watcher:127.0.0.1:38962 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 261040 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 261041 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 261043 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 261044 T1248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c1cfd name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 261045 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 261047 T1190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 262049 T1190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48187_dboil%2Fb
   [junit4]   2> 262050 T1190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48187_dboil%2Fb
   [junit4]   2> 262051 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 262052 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 262053 T1212 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 262053 T1248 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 262053 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 262053 T1219 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 262054 T1232 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 262058 T1249 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 262058 T1249 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 262060 T1249 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 262371 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 262372 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33184_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33184/dboil/b"}
   [junit4]   2> 262376 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48187_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48187/dboil/b"}
   [junit4]   2> 262376 T1213 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 262377 T1213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 262380 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 262380 T1248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 262380 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 262380 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 263060 T1249 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 263060 T1249 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/collection1
   [junit4]   2> 263061 T1249 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 263061 T1249 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 263062 T1249 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 263063 T1249 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/collection1/'
   [junit4]   2> 263064 T1249 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/collection1/lib/README' to classloader
   [junit4]   2> 263064 T1249 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/collection1/lib/classes/' to classloader
   [junit4]   2> 263119 T1249 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 263156 T1249 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 263158 T1249 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 263164 T1249 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 263579 T1249 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 263580 T1249 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 263580 T1249 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 263586 T1249 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 263588 T1249 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 263598 T1249 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 263600 T1249 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 263603 T1249 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 263604 T1249 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 263604 T1249 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 263604 T1249 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 263605 T1249 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 263605 T1249 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 263605 T1249 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 263606 T1249 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.SyncSliceTest-jetty2-1376841118046/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty2/
   [junit4]   2> 263606 T1249 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092c6
   [junit4]   2> 263607 T1249 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty2
   [junit4]   2> 263607 T1249 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty2/index/
   [junit4]   2> 263607 T1249 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 263607 T1249 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty2/index
   [junit4]   2> 263608 T1249 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 263608 T1249 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c0f08d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1282409),segFN=segments_1,generation=1}
   [junit4]   2> 263609 T1249 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 263612 T1249 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 263612 T1249 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 263612 T1249 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 263612 T1249 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 263613 T1249 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 263613 T1249 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 263613 T1249 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 263613 T1249 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 263613 T1249 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 263614 T1249 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 263614 T1249 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 263614 T1249 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 263615 T1249 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 263615 T1249 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 263615 T1249 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 263622 T1249 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 263624 T1249 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 263624 T1249 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 263625 T1249 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7c7fbd
   [junit4]   2> 263625 T1249 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c0f08d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1282409),segFN=segments_1,generation=1}
   [junit4]   2> 263625 T1249 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 263626 T1249 oass.SolrIndexSearcher.<init> Opening Searcher@1824007 main
   [junit4]   2> 263628 T1250 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1824007 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 263630 T1249 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 263630 T1249 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48187/dboil/b collection:collection1 shard:shard1
   [junit4]   2> 263630 T1249 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 263634 T1249 oasc.ZkController.register We are http://127.0.0.1:48187/dboil/b/collection1/ and leader is http://127.0.0.1:33184/dboil/b/collection1/
   [junit4]   2> 263634 T1249 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48187/dboil/b
   [junit4]   2> 263634 T1249 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 263634 T1249 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C1680 name=collection1 org.apache.solr.core.SolrCore@a40859 url=http://127.0.0.1:48187/dboil/b/collection1 node=127.0.0.1:48187_dboil%2Fb C1680_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:48187_dboil%2Fb, base_url=http://127.0.0.1:48187/dboil/b}
   [junit4]   2> 263634 T1251 C1680 P48187 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 263635 T1249 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 263635 T1251 C1680 P48187 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 263635 T1251 C1680 P48187 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 263635 T1251 C1680 P48187 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 263636 T1190 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 263636 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 263636 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 263638 T1251 C1680 P48187 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 263642 T1225 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 263706 T1190 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 263709 T1190 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57172
   [junit4]   2> 263709 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 263710 T1190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 263710 T1190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782
   [junit4]   2> 263710 T1190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/'
   [junit4]   2> 263720 T1190 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/solr.xml
   [junit4]   2> 263759 T1190 oasc.CoreContainer.<init> New CoreContainer 23637090
   [junit4]   2> 263759 T1190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/]
   [junit4]   2> 263760 T1190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 263760 T1190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 263761 T1190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 263761 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 263761 T1190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 263761 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 263762 T1190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 263762 T1190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 263762 T1190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 263762 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 263765 T1190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38962/solr
   [junit4]   2> 263765 T1190 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 263766 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 263767 T1263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fcd05 name:ZooKeeperConnection Watcher:127.0.0.1:38962 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 263767 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 263768 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 263771 T1190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 263772 T1265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a994e name:ZooKeeperConnection Watcher:127.0.0.1:38962/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 263772 T1190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 263775 T1190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 263886 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 263887 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48187_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48187/dboil/b"}
   [junit4]   2> 263892 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 263892 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 263892 T1265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 263892 T1248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 263892 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 264643 T1225 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 264643 T1225 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:48187_dboil%252Fb&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
   [junit4]   2> 264777 T1190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57172_dboil%2Fb
   [junit4]   2> 264778 T1190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57172_dboil%2Fb
   [junit4]   2> 264780 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 264780 T1265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 264781 T1212 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 264780 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 264780 T1248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 264781 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 264782 T1219 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 264782 T1265 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 264782 T1232 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 264783 T1248 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 264787 T1266 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 264787 T1266 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 264788 T1266 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 265402 T1213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 265403 T1213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57172_dboil%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57172/dboil/b"}
   [junit4]   2> 265403 T1213 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 265403 T1213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 265407 T1219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 265407 T1265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 265407 T1232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 265407 T1248 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 265407 T1212 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 265788 T1266 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 265789 T1266 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/collection1
   [junit4]   2> 265789 T1266 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 265791 T1266 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 265791 T1266 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 265793 T1266 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/collection1/'
   [junit4]   2> 265795 T1266 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/collection1/lib/README' to classloader
   [junit4]   2> 265795 T1266 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/collection1/lib/classes/' to classloader
   [junit4]   2> 265848 T1266 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 265882 T1266 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 265884 T1266 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 265889 T1266 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 266263 T1266 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 266263 T1266 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 266264 T1266 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 266269 T1266 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 266271 T1266 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 266284 T1266 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 266287 T1266 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 266290 T1266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 266291 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 266291 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 266291 T1266 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 266292 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 266292 T1266 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 266292 T1266 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 266293 T1266 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.SyncSliceTest-jetty3-1376841120782/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty3/
   [junit4]   2> 266293 T1266 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092c6
   [junit4]   2> 266293 T1266 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty3
   [junit4]   2> 266294 T1266 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty3/index/
   [junit4]   2> 266294 T1266 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 266294 T1266 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1376841111594/jetty3/index
   [junit4]   2> 266294 T1266 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=30, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 266295 T1266 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@614c27 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e660ee),segFN=segments_1,generation=1}
   [junit4]   2> 266295 T1266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 266298 T1266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 266298 T1266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 266298 T1266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 266298 T1266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 266299 T1266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 266299 T1266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 266299 T1266 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 266299 T1266 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 266300 T1266 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 266300 T1266 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 266301 T1266 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 266301 T1266 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 266301 T1266 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 266301 T1266 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 266302 T1266 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 266304 T1266 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 266306 T1266 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 266306 T1266 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 266306 T1266 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7022df
   [junit4]   2> 266307 T1266 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@614c27 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1e660ee),segFN=segments_1,generation=1}
   [junit4]   2> 266307 T1266 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 266307 T1266 oass.SolrIndexSearcher.<init> Opening Searcher@1a7b17e main
   [junit4]   2> 266310 T1267 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a7b17e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 266312 T1266 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 266312 T1266 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57172/dboil/b collection:collection1 shard:shard1
   [junit4]   2> 266312 T1266 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 266316 T1266 oasc.ZkController.register We are http://127.0.0.1:57172/dboil/b/collection1/ and leader is http://127.0.0.1:33184/dboil/b/collection1/
   [junit4]   2> 266317 T1266 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57172/dboil/b
   [junit4]   2> 266317 T1266 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 266317 T1266 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C1681 name=collection1 org.apache.solr.core.SolrCore@cae085 url=http://127.0.0.1:57172/dboil/b/collection1 node=127.0.0.1:57172_dboil%2Fb C1681_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:57172_dboil%2Fb, base_url=http://127.0.0.1:57172/dboil/b}
   [junit4]   2> 266317 T1268 C1681 P57172 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 266318 T1268 C1681 P57172 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 266318 T1266 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 266318 T1268 C1681 P57172 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 266318 T1268 C1681 P57172 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 266319 T1190 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 266319 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 266319 T1190 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 266320 T1268 C1681 P57172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 266323 T1227 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 266398 T1190 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 266400 T1190 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41835
   [junit4]   2> 266401 T1190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 266401 T1190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 266401 T1190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1376841123466
   [junit4]   2> 266402 T1190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1376841123466/'
   [junit4]   2> 266412 T1190 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1376841123466/solr.xml
   [junit4]   2> 266447 T1190 oasc.CoreContainer.<init> New CoreContainer 3907570
   [junit4]   2> 266447 T1190 oasc.CoreContainer.load Loading cores into CoreConta

[...truncated too long message...]

                  "node_name":"127.0.0.1:48187_dboil%2Fb",
   [junit4]   1>                   "base_url":"http://127.0.0.1:48187/dboil/b"},
   [junit4]   1>                 "core_node3":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:57172_dboil%2Fb",
   [junit4]   1>                   "base_url":"http://127.0.0.1:57172/dboil/b"},
   [junit4]   1>                 "core_node4":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:41835_dboil%2Fb",
   [junit4]   1>                   "base_url":"http://127.0.0.1:41835/dboil/b"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"},
   [junit4]   1>         "control_collection":{
   [junit4]   1>           "shards":{"shard1":{
   [junit4]   1>               "range":"80000000-7fffffff",
   [junit4]   1>               "state":"active",
   [junit4]   1>               "replicas":{"core_node1":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:33512_dboil%2Fb",
   [junit4]   1>                   "base_url":"http://127.0.0.1:33512/dboil/b",
   [junit4]   1>                   "leader":"true"}}}},
   [junit4]   1>           "maxShardsPerNode":"1",
   [junit4]   1>           "router":"compositeId",
   [junit4]   1>           "replicationFactor":"1"}}
   [junit4]   1>   /solr/aliases.json (0)
   [junit4]   1>   /solr/live_nodes (4)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:33184_dboil%2Fb (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:33512_dboil%2Fb (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:57172_dboil%2Fb (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:41835_dboil%2Fb (0)
   [junit4]   1>   /solr/overseer (3)
   [junit4]   1>   DATA:
   [junit4]   1>       
   [junit4]   1>    /solr/overseer/queue (0)
   [junit4]   1>    /solr/overseer/queue-work (0)
   [junit4]   1>    /solr/overseer/collection-queue-work (0)
   [junit4]   1>   /solr/collections (2)
   [junit4]   1>    /solr/collections/collection1 (3)
   [junit4]   1>    DATA:
   [junit4]   1>        {"configName":"conf1"}
   [junit4]   1>     /solr/collections/collection1/shards (0)
   [junit4]   1>     /solr/collections/collection1/leader_elect (1)
   [junit4]   1>      /solr/collections/collection1/leader_elect/shard1 (1)
   [junit4]   1>       /solr/collections/collection1/leader_elect/shard1/election (3)
   [junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/90232659089621006-core_node1-n_0000000004 (0)
   [junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/90232659089621004-core_node4-n_0000000006 (0)
   [junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/90232659089621002-core_node3-n_0000000005 (0)
   [junit4]   1>     /solr/collections/collection1/leaders (1)
   [junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "core":"collection1",
   [junit4]   1>            "node_name":"127.0.0.1:33184_dboil%2Fb",
   [junit4]   1>            "base_url":"http://127.0.0.1:33184/dboil/b"}
   [junit4]   1>    /solr/collections/control_collection (3)
   [junit4]   1>    DATA:
   [junit4]   1>        {"configName":"conf1"}
   [junit4]   1>     /solr/collections/control_collection/shards (0)
   [junit4]   1>     /solr/collections/control_collection/leader_elect (1)
   [junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
   [junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
   [junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/90232659089620995-core_node1-n_0000000000 (0)
   [junit4]   1>     /solr/collections/control_collection/leaders (1)
   [junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "core":"collection1",
   [junit4]   1>            "node_name":"127.0.0.1:33512_dboil%2Fb",
   [junit4]   1>            "base_url":"http://127.0.0.1:33512/dboil/b"}
   [junit4]   1>   /solr/overseer_elect (2)
   [junit4]   1>    /solr/overseer_elect/election (4)
   [junit4]   1>     /solr/overseer_elect/election/90232659089620995-127.0.0.1:33512_dboil%2Fb-n_0000000000 (0)
   [junit4]   1>     /solr/overseer_elect/election/90232659089621002-127.0.0.1:57172_dboil%2Fb-n_0000000003 (0)
   [junit4]   1>     /solr/overseer_elect/election/90232659089621006-127.0.0.1:33184_dboil%2Fb-n_0000000005 (0)
   [junit4]   1>     /solr/overseer_elect/election/90232659089621004-127.0.0.1:41835_dboil%2Fb-n_0000000004 (0)
   [junit4]   1>    /solr/overseer_elect/leader (0)
   [junit4]   1>    DATA:
   [junit4]   1>        {"id":"90232659089620995-127.0.0.1:33512_dboil%2Fb-n_0000000000"}
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=F9893A201817B35E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ro_RO -Dtests.timezone=SystemV/MST7 -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 98.6s J0 | SyncSliceTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:57172/dboil/b/collection1lastClient and got 5 from http://127.0.0.1:41835/dboil/b/collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F9893A201817B35E:786FB4386F48D362]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1128)
   [junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 353077 T1190 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 98635 T1189 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 353997 T1282 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 353997 T1282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 353998 T1282 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=Asserting), _version_=PostingsFormat(name=Direct), rnd_b=Pulsing41(freqCutoff=13 minBlockSize=62 maxBlockSize=204), intDefault=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Memory doPackFST= false), id=Pulsing41(freqCutoff=13 minBlockSize=62 maxBlockSize=204), a_t=PostingsFormat(name=Memory doPackFST= false), range_facet_sl=Pulsing41(freqCutoff=13 minBlockSize=62 maxBlockSize=204), range_facet_si=PostingsFormat(name=Memory doPackFST= false), other_tl1=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=Memory doPackFST= false), a_si=PostingsFormat(name=Direct)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ro_RO, timezone=SystemV/MST7
   [junit4]   2> NOTE: Linux 3.2.0-51-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=6,free=16768248,total=114212864
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressVersions, TestBadConfig, HdfsBasicDistributedZk2Test, TestConfig, SliceStateUpdateTest, TestSolrXMLSerializer, HdfsUnloadDistributedZkTest, LeaderElectionTest, QueryElevationComponentTest, TestCoreContainer, TestShardHandlerFactory, SliceStateTest, SuggesterFSTTest, TestCharFilters, SearchHandlerTest, TestJmxMonitoredMap, SolrInfoMBeanTest, TestValueSourceCache, MoreLikeThisHandlerTest, CSVRequestHandlerTest, TestSolrXml, ShardRoutingTest, TestPerFieldSimilarity, TestElisionMultitermQuery, TestSolrXmlPersistence, TestFieldTypeCollectionResource, BasicDistributedZkTest, PolyFieldTest, TestImplicitCoreProperties, IndexSchemaTest, DistributedQueryElevationComponentTest, SyncSliceTest]
   [junit4] Completed on J0 in 99.67s, 1 test, 1 failure <<< FAILURES!

[...truncated 770 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 323 suites, 1376 tests, 1 failure, 35 ignored (7 assumptions)

Total time: 48 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure