You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/20 06:25:23 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b93) - Build # 6219 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6219/
Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:52742/ba/l/collection1lastClient and got 252 from http://127.0.0.1:36784/ba/l/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:52742/ba/l/collection1lastClient and got 252 from http://127.0.0.1:36784/ba/l/collection1
	at __randomizedtesting.SeedInfo.seed([BC868A7A59F8CA9D:3D6004622EA7AAA1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9352 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 239144 T738 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ba/l
[junit4:junit4]   2> 239150 T738 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371701366133
[junit4:junit4]   2> 239151 T738 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 239151 T739 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 239252 T738 oasc.ZkTestServer.run start zk server on port:51018
[junit4:junit4]   2> 239252 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 239335 T745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fd022b8 name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 239335 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 239335 T738 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 239339 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 239344 T747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4117d532 name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 239344 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 239344 T738 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 239347 T738 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 239349 T738 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 239350 T738 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 239352 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 239353 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 239356 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 239356 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 239358 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 239359 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 239360 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 239361 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 239362 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 239363 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 239365 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 239365 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 239367 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 239367 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 239369 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 239369 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 239371 T738 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 239371 T738 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 239443 T738 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 239445 T738 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52988
[junit4:junit4]   2> 239446 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 239446 T738 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 239446 T738 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358
[junit4:junit4]   2> 239446 T738 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/solr.xml
[junit4:junit4]   2> 239446 T738 oasc.CoreContainer.<init> New CoreContainer 906064664
[junit4:junit4]   2> 239447 T738 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/'
[junit4:junit4]   2> 239447 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/'
[junit4:junit4]   2> 239488 T738 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 239488 T738 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 239488 T738 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 239489 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 239489 T738 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 239489 T738 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 239489 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 239490 T738 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 239490 T738 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 239490 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 239494 T738 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 239494 T738 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51018/solr
[junit4:junit4]   2> 239494 T738 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 239495 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 239496 T758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a8724cd name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 239497 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 239498 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 239501 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 239503 T760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19af7673 name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 239503 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 239504 T738 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 239506 T738 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 239508 T738 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 239512 T738 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52988_ba%2Fl
[junit4:junit4]   2> 239514 T738 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52988_ba%2Fl
[junit4:junit4]   2> 239517 T738 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 239521 T738 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 239527 T738 oasc.Overseer.start Overseer (id=89895820731023363-127.0.0.1:52988_ba%2Fl-n_0000000000) starting
[junit4:junit4]   2> 239529 T738 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 239532 T762 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 239533 T738 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 239535 T738 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 239537 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 239539 T761 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 239541 T763 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 239542 T763 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 241042 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 241043 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:52988_ba%2Fl",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:52988/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 241043 T761 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 241043 T761 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 241046 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 241543 T763 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/collection1
[junit4:junit4]   2> 241543 T763 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 241544 T763 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 241544 T763 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 241545 T763 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/collection1/'
[junit4:junit4]   2> 241546 T763 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/collection1/lib/README' to classloader
[junit4:junit4]   2> 241546 T763 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 241588 T763 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 241632 T763 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 241634 T763 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 241649 T763 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 241989 T763 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 241997 T763 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 241999 T763 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 242014 T763 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 242017 T763 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 242019 T763 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 242020 T763 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 242020 T763 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 242021 T763 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 242021 T763 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 242022 T763 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 242022 T763 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 242028 T763 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371701366358/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/
[junit4:junit4]   2> 242028 T763 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c11bf6
[junit4:junit4]   2> 242029 T763 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 242029 T763 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data
[junit4:junit4]   2> 242029 T763 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index/
[junit4:junit4]   2> 242030 T763 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 242030 T763 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index
[junit4:junit4]   2> 242033 T763 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 242033 T763 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 242035 T763 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 242035 T763 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 242035 T763 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 242036 T763 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 242036 T763 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 242036 T763 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 242037 T763 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 242037 T763 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 242037 T763 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 242039 T763 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 242042 T763 oass.SolrIndexSearcher.<init> Opening Searcher@496e30d5 main
[junit4:junit4]   2> 242043 T763 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 242043 T763 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 242047 T764 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@496e30d5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 242048 T763 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 242048 T763 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52988/ba/l collection:control_collection shard:shard1
[junit4:junit4]   2> 242049 T763 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 242054 T763 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 242056 T763 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 242056 T763 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 242056 T763 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52988/ba/l/collection1/
[junit4:junit4]   2> 242056 T763 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 242056 T763 oasc.SyncStrategy.syncToMe http://127.0.0.1:52988/ba/l/collection1/ has no replicas
[junit4:junit4]   2> 242056 T763 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52988/ba/l/collection1/
[junit4:junit4]   2> 242056 T763 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 242550 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 242556 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 242562 T763 oasc.ZkController.register We are http://127.0.0.1:52988/ba/l/collection1/ and leader is http://127.0.0.1:52988/ba/l/collection1/
[junit4:junit4]   2> 242562 T763 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52988/ba/l
[junit4:junit4]   2> 242562 T763 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 242563 T763 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 242563 T763 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 242565 T763 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 242567 T738 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 242567 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 242567 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 242570 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 242571 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 242573 T767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f00645f name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 242573 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 242574 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 242576 T738 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 242642 T738 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 242643 T738 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52742
[junit4:junit4]   2> 242645 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 242645 T738 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 242645 T738 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558
[junit4:junit4]   2> 242646 T738 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/solr.xml
[junit4:junit4]   2> 242646 T738 oasc.CoreContainer.<init> New CoreContainer 121059243
[junit4:junit4]   2> 242646 T738 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/'
[junit4:junit4]   2> 242646 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/'
[junit4:junit4]   2> 242690 T738 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 242691 T738 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 242691 T738 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 242691 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 242692 T738 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 242692 T738 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 242692 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 242693 T738 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 242693 T738 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 242693 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 242697 T738 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 242698 T738 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51018/solr
[junit4:junit4]   2> 242698 T738 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 242699 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 242700 T778 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cc8acd0 name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 242701 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 242702 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 242706 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 242707 T780 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60a88492 name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 242707 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 242710 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 243712 T738 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52742_ba%2Fl
[junit4:junit4]   2> 243714 T738 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52742_ba%2Fl
[junit4:junit4]   2> 243717 T760 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 243717 T767 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 243717 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 243718 T780 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 243724 T781 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 243724 T781 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 244060 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 244061 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:52988_ba%2Fl",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:52988/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 244063 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:52742_ba%2Fl",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:52742/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 244064 T761 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 244064 T761 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 244068 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 244068 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 244068 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 244725 T781 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/collection1
[junit4:junit4]   2> 244725 T781 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 244726 T781 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 244726 T781 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 244727 T781 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/collection1/'
[junit4:junit4]   2> 244728 T781 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/collection1/lib/README' to classloader
[junit4:junit4]   2> 244728 T781 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 244765 T781 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 244809 T781 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 244810 T781 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 244824 T781 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 245152 T781 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 245159 T781 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 245161 T781 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 245176 T781 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 245179 T781 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 245182 T781 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 245183 T781 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 245183 T781 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 245183 T781 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 245184 T781 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 245184 T781 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 245184 T781 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 245185 T781 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371701369558/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/
[junit4:junit4]   2> 245185 T781 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c11bf6
[junit4:junit4]   2> 245185 T781 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 245186 T781 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1
[junit4:junit4]   2> 245186 T781 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index/
[junit4:junit4]   2> 245186 T781 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 245186 T781 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index
[junit4:junit4]   2> 245189 T781 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 245189 T781 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 245192 T781 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 245192 T781 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 245193 T781 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 245193 T781 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 245194 T781 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 245194 T781 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 245194 T781 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 245195 T781 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 245195 T781 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 245197 T781 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 245200 T781 oass.SolrIndexSearcher.<init> Opening Searcher@5d59fb59 main
[junit4:junit4]   2> 245201 T781 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 245201 T781 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 245207 T782 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d59fb59 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 245210 T781 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 245210 T781 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52742/ba/l collection:collection1 shard:shard1
[junit4:junit4]   2> 245210 T781 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 245216 T781 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 245217 T781 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 245218 T781 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 245218 T781 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 245218 T781 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 245218 T781 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/ has no replicas
[junit4:junit4]   2> 245218 T781 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 245218 T781 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 245573 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 245578 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 245578 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 245578 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 245624 T781 oasc.ZkController.register We are http://127.0.0.1:52742/ba/l/collection1/ and leader is http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 245624 T781 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52742/ba/l
[junit4:junit4]   2> 245624 T781 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 245624 T781 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 245624 T781 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 245626 T781 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 245627 T738 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 245627 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 245628 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 245696 T738 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 245699 T738 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37291
[junit4:junit4]   2> 245700 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 245700 T738 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 245700 T738 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613
[junit4:junit4]   2> 245701 T738 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/solr.xml
[junit4:junit4]   2> 245701 T738 oasc.CoreContainer.<init> New CoreContainer 921713740
[junit4:junit4]   2> 245701 T738 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/'
[junit4:junit4]   2> 245701 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/'
[junit4:junit4]   2> 245746 T738 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 245746 T738 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 245747 T738 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 245747 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 245747 T738 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 245747 T738 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 245747 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 245748 T738 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 245748 T738 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 245748 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 245752 T738 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 245752 T738 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51018/solr
[junit4:junit4]   2> 245752 T738 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 245753 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 245754 T794 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b0bc25f name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 245755 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 245756 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 245760 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 245762 T796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6595870d name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 245762 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 245764 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 246767 T738 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37291_ba%2Fl
[junit4:junit4]   2> 246768 T738 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37291_ba%2Fl
[junit4:junit4]   2> 246769 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 246770 T760 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 246770 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 246770 T796 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 246770 T780 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 246770 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 246771 T767 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 246777 T797 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 246777 T797 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 247084 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 247085 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:52742_ba%2Fl",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:52742/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 247088 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37291_ba%2Fl",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37291/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 247088 T761 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 247088 T761 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 247091 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 247091 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 247091 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 247091 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 247779 T797 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/collection1
[junit4:junit4]   2> 247779 T797 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 247780 T797 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 247780 T797 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 247781 T797 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/collection1/'
[junit4:junit4]   2> 247781 T797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/collection1/lib/README' to classloader
[junit4:junit4]   2> 247782 T797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 247815 T797 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 247845 T797 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 247847 T797 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 247861 T797 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 248199 T797 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 248206 T797 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 248208 T797 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 248224 T797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 248227 T797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 248230 T797 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 248231 T797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 248232 T797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 248232 T797 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 248232 T797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 248233 T797 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 248233 T797 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 248233 T797 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371701372613/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/
[junit4:junit4]   2> 248233 T797 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c11bf6
[junit4:junit4]   2> 248234 T797 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 248234 T797 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2
[junit4:junit4]   2> 248234 T797 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index/
[junit4:junit4]   2> 248234 T797 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 248235 T797 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index
[junit4:junit4]   2> 248238 T797 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 248238 T797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 248240 T797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 248241 T797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 248241 T797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 248242 T797 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 248242 T797 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 248242 T797 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 248243 T797 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 248243 T797 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 248244 T797 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 248245 T797 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 248248 T797 oass.SolrIndexSearcher.<init> Opening Searcher@3a65c819 main
[junit4:junit4]   2> 248249 T797 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 248249 T797 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 248256 T798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a65c819 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 248260 T797 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 248260 T797 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37291/ba/l collection:collection1 shard:shard1
[junit4:junit4]   2> 248262 T797 oasc.ZkController.register We are http://127.0.0.1:37291/ba/l/collection1/ and leader is http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 248263 T797 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37291/ba/l
[junit4:junit4]   2> 248263 T797 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 248263 T797 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C1435 name=collection1 org.apache.solr.core.SolrCore@60c00196 url=http://127.0.0.1:37291/ba/l/collection1 node=127.0.0.1:37291_ba%2Fl C1435_STATE=coll:collection1 core:collection1 props:{collection=collection1, node_name=127.0.0.1:37291_ba%2Fl, state=down, base_url=http://127.0.0.1:37291/ba/l, core=collection1}
[junit4:junit4]   2> 248263 T799 C1435 P37291 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 248263 T797 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 248264 T799 C1435 P37291 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 248264 T799 C1435 P37291 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 248264 T799 C1435 P37291 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 248265 T738 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 248265 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 248266 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 248266 T799 C1435 P37291 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 248274 T774 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 248362 T738 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 248365 T738 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36784
[junit4:junit4]   2> 248365 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 248366 T738 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 248366 T738 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255
[junit4:junit4]   2> 248366 T738 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/solr.xml
[junit4:junit4]   2> 248366 T738 oasc.CoreContainer.<init> New CoreContainer 499906550
[junit4:junit4]   2> 248367 T738 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/'
[junit4:junit4]   2> 248367 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/'
[junit4:junit4]   2> 248409 T738 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 248409 T738 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 248409 T738 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 248410 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 248410 T738 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 248410 T738 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 248410 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 248410 T738 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 248411 T738 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 248411 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 248414 T738 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 248415 T738 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51018/solr
[junit4:junit4]   2> 248415 T738 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 248416 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 248417 T811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bac46a2 name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 248418 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 248419 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 248422 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 248424 T813 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1734c685 name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 248424 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 248426 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 248595 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 248596 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37291_ba%2Fl",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37291/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 248599 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 248599 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 248599 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 248599 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 248600 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 249274 T774 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 249274 T774 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&nodeName=127.0.0.1:37291_ba%252Fl&onlyIfLeader=true&state=recovering&checkLive=true&coreNodeName=3&wt=javabin&core=collection1} status=0 QTime=1000 
[junit4:junit4]   2> 249429 T738 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36784_ba%2Fl
[junit4:junit4]   2> 249430 T738 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36784_ba%2Fl
[junit4:junit4]   2> 249432 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 249432 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 249432 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 249432 T760 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 249432 T796 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 249432 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 249432 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 249434 T767 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 249434 T780 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 249434 T813 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 249439 T814 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 249439 T814 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 250103 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 250108 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36784_ba%2Fl",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36784/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 250108 T761 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 250108 T761 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 250112 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 250112 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 250112 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 250112 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 250112 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 250440 T814 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/collection1
[junit4:junit4]   2> 250441 T814 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 250441 T814 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 250441 T814 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 250442 T814 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/collection1/'
[junit4:junit4]   2> 250443 T814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/collection1/lib/README' to classloader
[junit4:junit4]   2> 250443 T814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 250471 T814 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 250505 T814 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 250506 T814 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 250522 T814 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 250861 T814 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 250870 T814 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 250876 T814 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 250892 T814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 250895 T814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 250898 T814 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 250899 T814 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 250899 T814 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 250900 T814 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 250900 T814 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 250900 T814 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 250901 T814 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 250901 T814 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371701375255/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/
[junit4:junit4]   2> 250901 T814 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c11bf6
[junit4:junit4]   2> 250901 T814 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 250902 T814 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3
[junit4:junit4]   2> 250902 T814 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index/
[junit4:junit4]   2> 250902 T814 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 250902 T814 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index
[junit4:junit4]   2> 250905 T814 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 250906 T814 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 250908 T814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 250908 T814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 250909 T814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 250909 T814 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 250910 T814 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 250910 T814 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 250910 T814 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 250911 T814 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 250911 T814 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 250913 T814 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 250916 T814 oass.SolrIndexSearcher.<init> Opening Searcher@66da12fe main
[junit4:junit4]   2> 250916 T814 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 250917 T814 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 250922 T815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66da12fe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 250925 T814 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 250925 T814 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36784/ba/l collection:collection1 shard:shard1
[junit4:junit4]   2> 250927 T814 oasc.ZkController.register We are http://127.0.0.1:36784/ba/l/collection1/ and leader is http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 250927 T814 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36784/ba/l
[junit4:junit4]   2> 250927 T814 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 250927 T814 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C1436 name=collection1 org.apache.solr.core.SolrCore@61346b3b url=http://127.0.0.1:36784/ba/l/collection1 node=127.0.0.1:36784_ba%2Fl C1436_STATE=coll:collection1 core:collection1 props:{collection=collection1, node_name=127.0.0.1:36784_ba%2Fl, state=down, base_url=http://127.0.0.1:36784/ba/l, core=collection1}
[junit4:junit4]   2> 250928 T816 C1436 P36784 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 250928 T816 C1436 P36784 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 250928 T814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 250928 T816 C1436 P36784 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 250928 T816 C1436 P36784 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 250929 T738 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 250929 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 250929 T816 C1436 P36784 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 250930 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 250937 T775 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 251005 T738 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 251008 T738 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45463
[junit4:junit4]   2> 251008 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 251009 T738 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 251009 T738 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917
[junit4:junit4]   2> 251009 T738 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/solr.xml
[junit4:junit4]   2> 251009 T738 oasc.CoreContainer.<init> New CoreContainer 824636527
[junit4:junit4]   2> 251010 T738 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/'
[junit4:junit4]   2> 251010 T738 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/'
[junit4:junit4]   2> 251057 T738 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 251057 T738 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 251058 T738 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 251058 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 251058 T738 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 251059 T738 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 251059 T738 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 251059 T738 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 251059 T738 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 251060 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 251065 T738 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 251066 T738 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51018/solr
[junit4:junit4]   2> 251066 T738 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 251067 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 251069 T828 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2191c237 name:ZooKeeperConnection Watcher:127.0.0.1:51018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 251069 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 251071 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 251076 T738 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 251077 T830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@711da56a name:ZooKeeperConnection Watcher:127.0.0.1:51018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 251077 T738 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 251080 T738 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C1437 name=collection1 org.apache.solr.core.SolrCore@60c00196 url=http://127.0.0.1:37291/ba/l/collection1 node=127.0.0.1:37291_ba%2Fl C1437_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:37291_ba%2Fl, state=recovering, base_url=http://127.0.0.1:37291/ba/l, core=collection1}
[junit4:junit4]   2> 251275 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52742/ba/l/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 251276 T799 C1437 P37291 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37291/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 251276 T799 C1437 P37291 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 251276 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 251276 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 251277 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 251277 T799 C1437 P37291 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 251277 T799 C1437 P37291 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52742/ba/l/collection1/. core=collection1
[junit4:junit4]   2> 251277 T799 C1437 P37291 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C1438 name=collection1 org.apache.solr.core.SolrCore@48553d3 url=http://127.0.0.1:52742/ba/l/collection1 node=127.0.0.1:52742_ba%2Fl C1438_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:52742_ba%2Fl, state=active, base_url=http://127.0.0.1:52742/ba/l, core=collection1, leader=true}
[junit4:junit4]   2> 251287 T776 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 251288 T774 C1438 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 251290 T774 C1438 P52742 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 251290 T774 C1438 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 251303 T774 C1438 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 251303 T774 C1438 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 251304 T774 C1438 P52742 oass.SolrIndexSearcher.<init> Opening Searcher@8e9973d realtime
[junit4:junit4]   2> 251304 T774 C1438 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 251305 T774 C1438 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin&openSearcher=false&commit_end_point=true} {commit=} 0 18
[junit4:junit4]   2> 251306 T799 C1437 P37291 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 251306 T799 C1437 P37291 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 251308 T773 C1438 P52742 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 251308 T773 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 251309 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 251309 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 251309 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 251311 T774 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&generation=2&command=filelist&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 251312 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 251312 T799 C1437 P37291 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294
[junit4:junit4]   2> 251313 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e769650 fullCopy=false
[junit4:junit4]   2> 251315 T776 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={checksum=true&generation=2&command=filecontent&file=segments_2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 251316 T799 C1437 P37291 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 251324 T799 C1437 P37291 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 251325 T799 C1437 P37291 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 251329 T799 C1437 P37291 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 251329 T799 C1437 P37291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 251329 T799 C1437 P37291 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 251330 T799 C1437 P37291 oass.SolrIndexSearcher.<init> Opening Searcher@1d12e1e6 main
[junit4:junit4]   2> 251331 T798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d12e1e6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 251331 T799 C1437 P37291 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294;done=true>>]
[junit4:junit4]   2> 251332 T799 C1437 P37291 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294
[junit4:junit4]   2> 251332 T799 C1437 P37291 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index.20130619190938294
[junit4:junit4]   2> 251333 T799 C1437 P37291 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 251333 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 251333 T799 C1437 P37291 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 251333 T799 C1437 P37291 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 251335 T799 C1437 P37291 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 251616 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 251617 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36784_ba%2Fl",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36784/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 251620 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37291_ba%2Fl",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37291/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 251643 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251644 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251644 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251644 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251644 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251643 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 251937 T775 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 251937 T775 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&nodeName=127.0.0.1:36784_ba%252Fl&onlyIfLeader=true&state=recovering&checkLive=true&coreNodeName=4&wt=javabin&core=collection1} status=0 QTime=1001 
[junit4:junit4]   2> 252082 T738 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45463_ba%2Fl
[junit4:junit4]   2> 252084 T738 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45463_ba%2Fl
[junit4:junit4]   2> 252085 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 252086 T813 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252086 T767 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252086 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 252086 T830 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252086 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 252086 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 252086 T780 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252086 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 252087 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 252087 T796 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252088 T760 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 252092 T833 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 252092 T833 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 253149 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 253150 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45463_ba%2Fl",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45463/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 253150 T761 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 253150 T761 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 253153 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 253153 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 253153 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 253153 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 253153 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 253153 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> ASYNC  NEW_CORE C1439 name=collection1 org.apache.solr.core.SolrCore@61346b3b url=http://127.0.0.1:36784/ba/l/collection1 node=127.0.0.1:36784_ba%2Fl C1439_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:36784_ba%2Fl, state=recovering, base_url=http://127.0.0.1:36784/ba/l, core=collection1}
[junit4:junit4]   2> 253938 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52742/ba/l/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 253938 T816 C1439 P36784 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36784/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 253938 T816 C1439 P36784 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 253939 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 253939 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 253939 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 253939 T816 C1439 P36784 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 253940 T816 C1439 P36784 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52742/ba/l/collection1/. core=collection1
[junit4:junit4]   2> 253940 T816 C1439 P36784 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 253941 T775 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 253945 T773 C1438 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 253953 T773 C1438 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 253953 T773 C1438 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 253954 T773 C1438 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 253954 T773 C1438 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin&openSearcher=false&commit_end_point=true} {commit=} 0 9
[junit4:junit4]   2> 253955 T816 C1439 P36784 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 253955 T816 C1439 P36784 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 253957 T775 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 253957 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 253957 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 253958 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 253960 T776 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&generation=3&command=filelist&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 253960 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 253961 T816 C1439 P36784 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942
[junit4:junit4]   2> 253961 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942 lockFactory=org.apache.lucene.store.NativeFSLockFactory@227c30e3 fullCopy=false
[junit4:junit4]   2> 253963 T773 C1438 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={checksum=true&generation=3&command=filecontent&file=segments_3&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 253964 T816 C1439 P36784 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 253967 T816 C1439 P36784 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 253967 T816 C1439 P36784 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 253969 T816 C1439 P36784 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 253970 T816 C1439 P36784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 253970 T816 C1439 P36784 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 253970 T816 C1439 P36784 oass.SolrIndexSearcher.<init> Opening Searcher@2e2f6281 main
[junit4:junit4]   2> 253971 T815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e2f6281 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 253972 T816 C1439 P36784 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942;done=true>>]
[junit4:junit4]   2> 253972 T816 C1439 P36784 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942
[junit4:junit4]   2> 253972 T816 C1439 P36784 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index.20130619190940942
[junit4:junit4]   2> 253973 T816 C1439 P36784 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 253973 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 253973 T816 C1439 P36784 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 253973 T816 C1439 P36784 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 253993 T816 C1439 P36784 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 254094 T833 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/collection1
[junit4:junit4]   2> 254094 T833 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 254094 T833 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 254094 T833 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 254095 T833 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/collection1/'
[junit4:junit4]   2> 254096 T833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/collection1/lib/README' to classloader
[junit4:junit4]   2> 254096 T833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 254129 T833 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 254161 T833 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 254162 T833 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 254179 T833 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 254526 T833 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 254532 T833 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 254534 T833 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 254548 T833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 254550 T833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 254553 T833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 254554 T833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 254554 T833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 254554 T833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 254555 T833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 254555 T833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 254555 T833 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 254556 T833 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371701377917/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/
[junit4:junit4]   2> 254556 T833 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c11bf6
[junit4:junit4]   2> 254556 T833 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 254557 T833 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4
[junit4:junit4]   2> 254557 T833 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index/
[junit4:junit4]   2> 254557 T833 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 254557 T833 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index
[junit4:junit4]   2> 254561 T833 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 254561 T833 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 254569 T833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 254570 T833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 254570 T833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 254571 T833 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 254571 T833 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 254571 T833 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 254573 T833 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 254573 T833 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 254574 T833 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 254575 T833 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 254578 T833 oass.SolrIndexSearcher.<init> Opening Searcher@4135ee38 main
[junit4:junit4]   2> 254579 T833 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 254579 T833 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 254584 T836 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4135ee38 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 254587 T833 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 254587 T833 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45463/ba/l collection:collection1 shard:shard1
[junit4:junit4]   2> 254590 T833 oasc.ZkController.register We are http://127.0.0.1:45463/ba/l/collection1/ and leader is http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 254591 T833 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45463/ba/l
[junit4:junit4]   2> 254591 T833 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 254591 T833 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C1440 name=collection1 org.apache.solr.core.SolrCore@5f294221 url=http://127.0.0.1:45463/ba/l/collection1 node=127.0.0.1:45463_ba%2Fl C1440_STATE=coll:collection1 core:collection1 props:{collection=collection1, node_name=127.0.0.1:45463_ba%2Fl, state=down, base_url=http://127.0.0.1:45463/ba/l, core=collection1}
[junit4:junit4]   2> 254591 T837 C1440 P45463 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 254592 T837 C1440 P45463 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 254592 T837 C1440 P45463 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 254592 T837 C1440 P45463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 254593 T833 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 254594 T837 C1440 P45463 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 254594 T738 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 254594 T738 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 254595 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 254601 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 254601 T776 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 254602 T738 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 254602 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 254603 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 254658 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 254658 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"4",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36784_ba%2Fl",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36784/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 254661 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45463_ba%2Fl",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45463/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 254664 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 254664 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 254665 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 254665 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 254664 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 254665 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 255602 T776 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 255602 T776 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&nodeName=127.0.0.1:45463_ba%252Fl&onlyIfLeader=true&state=recovering&checkLive=true&coreNodeName=5&wt=javabin&core=collection1} status=0 QTime=1001 
[junit4:junit4]   2> 255604 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 256605 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C1441 name=collection1 org.apache.solr.core.SolrCore@5f294221 url=http://127.0.0.1:45463/ba/l/collection1 node=127.0.0.1:45463_ba%2Fl C1441_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:45463_ba%2Fl, state=recovering, base_url=http://127.0.0.1:45463/ba/l, core=collection1}
[junit4:junit4]   2> 257603 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52742/ba/l/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 257604 T837 C1441 P45463 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45463/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 257604 T837 C1441 P45463 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 257604 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 257605 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 257605 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 257605 T837 C1441 P45463 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 257605 T837 C1441 P45463 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52742/ba/l/collection1/. core=collection1
[junit4:junit4]   2> 257605 T837 C1441 P45463 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 257607 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C1442 name=collection1 org.apache.solr.core.SolrCore@48553d3 url=http://127.0.0.1:52742/ba/l/collection1 node=127.0.0.1:52742_ba%2Fl C1442_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:52742_ba%2Fl, state=active, base_url=http://127.0.0.1:52742/ba/l, core=collection1, leader=true}
[junit4:junit4]   2> 257607 T773 C1442 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 257615 T774 C1442 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 257623 T774 C1442 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 257624 T774 C1442 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 257624 T774 C1442 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 257625 T774 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin&openSearcher=false&commit_end_point=true} {commit=} 0 10
[junit4:junit4]   2> 257626 T837 C1441 P45463 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 257626 T837 C1441 P45463 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 257629 T776 C1442 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 257629 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 257630 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 257630 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 257633 T773 C1442 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={version=2&generation=4&command=filelist&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 257633 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 257635 T837 C1441 P45463 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616
[junit4:junit4]   2> 257635 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1435c94 fullCopy=false
[junit4:junit4]   2> 257638 T774 C1442 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/replication params={checksum=true&generation=4&command=filecontent&file=segments_4&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 257640 T837 C1441 P45463 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 257644 T837 C1441 P45463 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 257644 T837 C1441 P45463 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 257648 T837 C1441 P45463 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 257649 T837 C1441 P45463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 257649 T837 C1441 P45463 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 257650 T837 C1441 P45463 oass.SolrIndexSearcher.<init> Opening Searcher@138b5e main
[junit4:junit4]   2> 257652 T836 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@138b5e main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 257653 T837 C1441 P45463 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616;done=true>>]
[junit4:junit4]   2> 257653 T837 C1441 P45463 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616
[junit4:junit4]   2> 257654 T837 C1441 P45463 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index.20130619190944616
[junit4:junit4]   2> 257654 T837 C1441 P45463 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 257654 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 257655 T837 C1441 P45463 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 257655 T837 C1441 P45463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 257674 T837 C1441 P45463 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 257676 T761 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 257677 T761 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core_node_name":"5",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45463_ba%2Fl",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45463/ba/l",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "shard_range":null}
[junit4:junit4]   2> 257682 T780 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 257682 T813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 257682 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 257682 T767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 257682 T760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 257682 T796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 258609 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 258610 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C1443 name=collection1 org.apache.solr.core.SolrCore@32f8876b url=http://127.0.0.1:52988/ba/l/collection1 node=127.0.0.1:52988_ba%2Fl C1443_STATE=coll:control_collection core:collection1 props:{collection=control_collection, shard=shard1, node_name=127.0.0.1:52988_ba%2Fl, state=active, base_url=http://127.0.0.1:52988/ba/l, core=collection1, leader=true}
[junit4:junit4]   2> 258620 T754 C1443 P52988 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 258622 T754 C1443 P52988 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 258623 T754 C1443 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 258633 T754 C1443 P52988 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 258634 T754 C1443 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 258635 T754 C1443 P52988 oass.SolrIndexSearcher.<init> Opening Searcher@6566728e main
[junit4:junit4]   2> 258635 T754 C1443 P52988 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 258637 T764 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6566728e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 258638 T754 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin} {commit=} 0 19
[junit4:junit4]   2> ASYNC  NEW_CORE C1444 name=collection1 org.apache.solr.core.SolrCore@60c00196 url=http://127.0.0.1:37291/ba/l/collection1 node=127.0.0.1:37291_ba%2Fl C1444_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:37291_ba%2Fl, state=active, base_url=http://127.0.0.1:37291/ba/l, core=collection1}
[junit4:junit4]   2> 258649 T792 C1444 P37291 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52742/ba/l/collection1/, StdNode: http://127.0.0.1:37291/ba/l/collection1/, StdNode: http://127.0.0.1:36784/ba/l/collection1/, StdNode: http://127.0.0.1:45463/ba/l/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 258653 T774 C1442 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 258653 T790 C1444 P37291 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2>  C1441_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:45463_ba%2Fl, state=active, base_url=http://127.0.0.1:45463/ba/l, core=collection1}
[junit4:junit4]   2> 258656 T825 C1441 P45463 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C1445 name=collection1 org.apache.solr.core.SolrCore@61346b3b url=http://127.0.0.1:36784/ba/l/collection1 node=127.0.0.1:36784_ba%2Fl C1445_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:36784_ba%2Fl, state=active, base_url=http://127.0.0.1:36784/ba/l, core=collection1}
[junit4:junit4]   2> 258658 T809 C1445 P36784 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 258673 T774 C1442 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 258674 T774 C1442 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 258674 T774 C1442 P52742 oass.SolrIndexSearcher.<init> Opening Searcher@715de19a main
[junit4:junit4]   2> 258674 T774 C1442 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 258675 T782 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@715de19a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 258676 T774 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 24
[junit4:junit4]   2> 258703 T790 C1444 P37291 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 258703 T825 C1441 P45463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 258704 T825 C1441 P45463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 258703 T809 C1445 P36784 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 258705 T809 C1445 P36784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 258705 T825 C1441 P45463 oass.SolrIndexSearcher.<init> Opening Searcher@f45201d main
[junit4:junit4]   2> 258704 T790 C1444 P37291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 258706 T809 C1445 P36784 oass.SolrIndexSearcher.<init> Opening Searcher@635f7017 main
[junit4:junit4]   2> 258706 T825 C1441 P45463 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 258707 T809 C1445 P36784 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 258707 T790 C1444 P37291 oass.SolrIndexSearcher.<init> Opening Searcher@2a25d662 main
[junit4:junit4]   2> 258708 T790 C1444 P37291 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 258709 T836 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 
[junit4:junit4]   2> 258708 T815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@635f7017 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 258709 T836 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f45201d main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 258710 T798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a25d662 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 258710 T809 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 52
[junit4:junit4]   2> 258711 T790 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 59
[junit4:junit4]   2> 258710 T825 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 54
[junit4:junit4]   2> 258712 T792 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin} {commit=} 0 63
[junit4:junit4]   2> 258714 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 258718 T776 C1442 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 258722 T789 C1444 P37291 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 258726 T808 C1445 P36784 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 258729 T824 C1441 P45463 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 260742 T754 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1438333154324643840)} 0 9
[junit4:junit4]   2> 260757 T789 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&update.from=http://127.0.0.1:52742/ba/l/collection1/&_version_=-1438333154337226752&wt=javabin} {deleteByQuery=*:* (-1438333154337226752)} 0 1
[junit4:junit4]   2> 260758 T808 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&update.from=http://127.0.0.1:52742/ba/l/collection1/&_version_=-1438333154337226752&wt=javabin} {deleteByQuery=*:* (-1438333154337226752)} 0 2
[junit4:junit4]   2> 260758 T826 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&update.from=http://127.0.0.1:52742/ba/l/collection1/&_version_=-1438333154337226752&wt=javabin} {deleteByQuery=*:* (-1438333154337226752)} 0 2
[junit4:junit4]   2> 260759 T773 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1438333154337226752)} 0 14
[junit4:junit4]   2> 260764 T756 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[0 (1438333154355052544)]} 0 2
[junit4:junit4]   2> 260772 T807 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[0 (1438333154360295424)]} 0 2
[junit4:junit4]   2> 260773 T825 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[0 (1438333154360295424)]} 0 3
[junit4:junit4]   2> 260773 T791 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[0 (1438333154360295424)]} 0 3
[junit4:junit4]   2> 260774 T774 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[0 (1438333154360295424)]} 0 7
[junit4:junit4]   2> 260777 T753 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[1 (1438333154369732608)]} 0 1
[junit4:junit4]   2> 260784 T824 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[1 (1438333154373926912)]} 0 1
[junit4:junit4]   2> 260785 T790 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[1 (1438333154373926912)]} 0 2
[junit4:junit4]   2> 260785 T809 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&wt=javabin} {add=[1 (1438333154373926912)]} 0 2
[junit4:junit4]   2> 260786 T775 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=TOLEADER&version=2&distrib.from=http://127.0.0.1:45463/ba/l/collection1/&wt=javabin} {add=[1 (1438333154373926912)]} 0 6
[junit4:junit4]   2> 260786 T823 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[1]} 0 8
[junit4:junit4]   2> 260791 T755 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[2 (1438333154383364096)]} 0 2
[junit4:junit4]   2> 260795 T776 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:37291/ba/l/collection1/]
[junit4:junit4]   2> 260796 T776 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37291/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:true
[junit4:junit4]   2> 260796 T776 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:36784/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:false
[junit4:junit4]   2> 260796 T776 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45463/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:false
[junit4:junit4]   2> 260800 T806 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[2 (1438333154391752704)]} 0 1
[junit4:junit4]   2> 260800 T826 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[2 (1438333154391752704)]} 0 1
[junit4:junit4]   2> 260801 T776 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=TOLEADER&version=2&distrib.from=http://127.0.0.1:37291/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[2 (1438333154391752704)]} 0 6
[junit4:junit4]   2> 260802 T792 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[2]} 0 9
[junit4:junit4]   2> 260805 T754 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&wt=javabin} {add=[3 (1438333154399092736)]} 0 1
[junit4:junit4]   2> 260810 T773 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:37291/ba/l/collection1/]
[junit4:junit4]   2> 260811 T773 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37291/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:true
[junit4:junit4]   2> 260811 T773 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:36784/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:false
[junit4:junit4]   2> 260811 T773 C1442 P52742 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45463/ba/l/collection1/ against:[http://127.0.0.1:37291/ba/l/collection1/] result:false
[junit4:junit4]   2> 260816 T825 C1441 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[3 (1438333154407481344)]} 0 1
[junit4:junit4]   2> 260816 T808 C1445 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=FROMLEADER&version=2&distrib.from=http://127.0.0.1:52742/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[3 (1438333154407481344)]} 0 1
[junit4:junit4]   2> 260818 T773 C1442 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={update.distrib=TOLEADER&version=2&distrib.from=http://127.0.0.1:37291/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&wt=javabin} {add=[3 (1438333154407481344)]} 0 8
[junit4:junit4]   2> 260818 T791 C1444 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={version=2&test.distrib.skip.servers=http://127.0.0.1:37291/ba/l/collection1/&test.distrib.skip.servers=http://127.0.0.1:36784/ba/l/collection1/&wt=javabin} {add=[3]} 0 11
[junit4:junit4]   2> 260822 T756 C1443 P52988 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 260853 T756 C1443 P52988 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 260854 T756 C1443 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 260860 T756 C1443 P52988 oass.SolrIndexSearcher.<init> Opening Searcher@7fdd5a32 main
[junit4:junit4]   2> 260861 T756 C1443 P52988 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 260862 T764 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7fdd5a32 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 260862 T756 C1443 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin} {commit=} 0 41
[junit4:junit4]   2> 260864 T774 C1442 P52742 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52742/ba/l/collection1/, StdNode: http://127.0.0.1:37291/ba/l/collection1/, StdNode: http://127.0.0.1:36784/ba/l/collection1/, StdNode: http://127.0.0.1:45463/ba/l/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C1446 name=collection1 org.apache.solr.core.SolrCore@48553d3 url=http://127.0.0.1:52742/ba/l/collection1 node=127.0.0.1:52742_ba%2Fl C1446_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:52742_ba%2Fl, state=active, base_url=http://127.0.0.1:52742/ba/l, core=collection1, leader=true}
[junit4:junit4]   2> 260869 T775 C1446 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C1447 name=collection1 org.apache.solr.core.SolrCore@60c00196 url=http://127.0.0.1:37291/ba/l/collection1 node=127.0.0.1:37291_ba%2Fl C1447_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:37291_ba%2Fl, state=active, base_url=http://127.0.0.1:37291/ba/l, core=collection1}
[junit4:junit4]   2> 260870 T790 C1447 P37291 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C1448 name=collection1 org.apache.solr.core.SolrCore@5f294221 url=http://127.0.0.1:45463/ba/l/collection1 node=127.0.0.1:45463_ba%2Fl C1448_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:45463_ba%2Fl, state=active, base_url=http://127.0.0.1:45463/ba/l, core=collection1}
[junit4:junit4]   2> 260871 T824 C1448 P45463 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C1449 name=collection1 org.apache.solr.core.SolrCore@61346b3b url=http://127.0.0.1:36784/ba/l/collection1 node=127.0.0.1:36784_ba%2Fl C1449_STATE=coll:collection1 core:collection1 props:{collection=collection1, shard=shard1, node_name=127.0.0.1:36784_ba%2Fl, state=active, base_url=http://127.0.0.1:36784/ba/l, core=collection1}
[junit4:junit4]   2> 260871 T807 C1449 P36784 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 260903 T790 C1447 P37291 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 260904 T790 C1447 P37291 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 260911 T790 C1447 P37291 oass.SolrIndexSearcher.<init> Opening Searcher@21b7fa0d main
[junit4:junit4]   2> 260912 T790 C1447 P37291 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 260913 T798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21b7fa0d main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 260913 T790 C1447 P37291 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 43
[junit4:junit4]   2> 260923 T824 C1448 P45463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 260923 T775 C1446 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 260924 T775 C1446 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 260923 T807 C1449 P36784 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 260924 T824 C1448 P45463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 260924 T807 C1449 P36784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 260927 T775 C1446 P52742 oass.SolrIndexSearcher.<init> Opening Searcher@6adee1eb main
[junit4:junit4]   2> 260928 T824 C1448 P45463 oass.SolrIndexSearcher.<init> Opening Searcher@f3ba02e main
[junit4:junit4]   2> 260928 T775 C1446 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 260928 T807 C1449 P36784 oass.SolrIndexSearcher.<init> Opening Searcher@2e1ef9d9 main
[junit4:junit4]   2> 260929 T836 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 
[junit4:junit4]   2> 260929 T782 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6adee1eb main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 260928 T824 C1448 P45463 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 260929 T807 C1449 P36784 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 260930 T775 C1446 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 61
[junit4:junit4]   2> 260930 T836 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f3ba02e main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 260930 T815 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 
[junit4:junit4]   2> 260931 T824 C1448 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 60
[junit4:junit4]   2> 260931 T815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e1ef9d9 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 260932 T807 C1449 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 62
[junit4:junit4]   2> 260932 T774 C1446 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin} {commit=} 0 68
[junit4:junit4]   2> 260933 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 260933 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 260934 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 260936 T776 C1446 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 260938 T789 C1447 P37291 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 260940 T809 C1449 P36784 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 260943 T823 C1448 P45463 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/select params={rows=0&version=2&q=*:*&tests=checkShardConsistency&distrib=false&wt=javabin} hits=4 status=0 QTime=1 
[junit4:junit4]   2> 260944 T738 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 260950 T806 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : collection=collection1&action=SYNCSHARD&version=2&shard=shard1&wt=javabin
[junit4:junit4]   2> 260950 T806 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 260954 T773 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 260954 T773 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 260955 T773 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l START replicas=[http://127.0.0.1:37291/ba/l/collection1/, http://127.0.0.1:36784/ba/l/collection1/, http://127.0.0.1:45463/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 260956 T792 C1447 P37291 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 260956 T826 C1448 P45463 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 260956 T808 C1449 P36784 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 260957 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Received 3 versions from 127.0.0.1:37291/ba/l/collection1/
[junit4:junit4]   2> 260957 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Our versions are newer. ourLowThreshold=1438333154337226752 otherHigh=1438333154373926912
[junit4:junit4]   2> 260957 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Received 5 versions from 127.0.0.1:45463/ba/l/collection1/
[junit4:junit4]   2> 260958 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Our versions are newer. ourLowThreshold=1438333154337226752 otherHigh=1438333154391752704
[junit4:junit4]   2> 260958 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Received 5 versions from 127.0.0.1:36784/ba/l/collection1/
[junit4:junit4]   2> 260958 T773 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l  Our versions are newer. ourLowThreshold=1438333154337226752 otherHigh=1438333154391752704
[junit4:junit4]   2> 260958 T773 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52742/ba/l DONE. sync succeeded
[junit4:junit4]   2> 260958 T773 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 260958 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/: try and ask http://127.0.0.1:37291/ba/l/collection1/ to sync
[junit4:junit4]   2> 260959 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/: try and ask http://127.0.0.1:36784/ba/l/collection1/ to sync
[junit4:junit4]   2> 260959 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/: try and ask http://127.0.0.1:45463/ba/l/collection1/ to sync
[junit4:junit4]   2> 260961 T791 C1447 P37291 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37291/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 260961 T807 C1449 P36784 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36784/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 260961 T825 C1448 P45463 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45463/ba/l START replicas=[http://127.0.0.1:52742/ba/l/collection1/] nUpdates=100
[junit4:junit4]   2> 260963 T775 C1446 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 260963 T774 C1446 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 260963 T776 C1446 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 260963 T825 C1448 P45463 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45463/ba/l  Received 5 versions from 127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 260963 T807 C1449 P36784 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36784/ba/l  Received 5 versions from 127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 260964 T825 C1448 P45463 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45463/ba/l  Our versions are newer. ourLowThreshold=1438333154337226752 otherHigh=1438333154391752704
[junit4:junit4]   2> 260964 T825 C1448 P45463 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45463/ba/l DONE. sync succeeded
[junit4:junit4]   2> 260964 T791 C1447 P37291 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:37291/ba/l  Received 5 versions from 127.0.0.1:52742/ba/l/collection1/
[junit4:junit4]   2> 260964 T825 C1448 P45463 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&sync=http://127.0.0.1:52742/ba/l/collection1/&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=3 
[junit4:junit4]   2> 260964 T807 C1449 P36784 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36784/ba/l  Our versions are newer. ourLowThreshold=1438333154337226752 otherHigh=1438333154391752704
[junit4:junit4]   2> 260965 T807 C1449 P36784 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36784/ba/l DONE. sync succeeded
[junit4:junit4]   2> 260965 T807 C1449 P36784 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&sync=http://127.0.0.1:52742/ba/l/collection1/&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=4 
[junit4:junit4]   2> 260964 T791 C1447 P37291 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:37291/ba/l Requesting updates from 127.0.0.1:52742/ba/l/collection1/n=2 versions=[1438333154407481344, 1438333154391752704]
[junit4:junit4]   2> 260965 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/:  sync completed with http://127.0.0.1:45463/ba/l/collection1/
[junit4:junit4]   2> 260966 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/:  sync completed with http://127.0.0.1:36784/ba/l/collection1/
[junit4:junit4]   2> 260968 T775 C1446 P52742 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&qt=/get&distrib=false&getUpdates=1438333154407481344,1438333154391752704&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 260972 T791 C1447 P37291 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438333154391752704), 3 (1438333154407481344)]} 0 2
[junit4:junit4]   2> 260973 T791 C1447 P37291 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37291/ba/l DONE. sync succeeded
[junit4:junit4]   2> 260973 T791 C1447 P37291 oasc.SolrCore.execute [collection1] webapp=/ba/l path=/get params={version=2&sync=http://127.0.0.1:52742/ba/l/collection1/&getVersions=100&qt=/get&distrib=false&wt=javabin} status=0 QTime=13 
[junit4:junit4]   2> 260974 T773 oasc.SyncStrategy.syncToMe http://127.0.0.1:52742/ba/l/collection1/:  sync completed with http://127.0.0.1:37291/ba/l/collection1/
[junit4:junit4]   2> 260974 T773 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={collection=collection1&action=REQUESTSYNCSHARD&version=2&shard=shard1&wt=javabin&core=collection1} status=0 QTime=20 
[junit4:junit4]   2> 260974 T806 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={collection=collection1&action=SYNCSHARD&version=2&shard=shard1&wt=javabin} status=0 QTime=25 
[junit4:junit4]   2> 260975 T738 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 260975 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 260976 T738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 260976 T738 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C1450 name=collection1 org.apache.solr.core.SolrCore@32f8876b url=http://127.0.0.1:52988/ba/l/collection1 node=127.0.0.1:52988_ba%2Fl C1450_STATE=coll:control_collection core:collection1 props:{collection=control_collection, shard=shard1, node_name=127.0.0.1:52988_ba%2Fl, state=active, base_url=http://127.0.0.1:52988/ba/l, core=collection1, leader=true}
[junit4:junit4]   2> 260978 T753 C1450 P52988 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 261023 T753 C1450 P52988 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/control/data/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 261024 T753 C1450 P52988 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 261024 T753 C1450 P52988 oass.SolrIndexSearcher.<init> Opening Searcher@3fee4d40 main
[junit4:junit4]   2> 261025 T753 C1450 P52988 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 261026 T764 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fee4d40 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 261027 T753 C1450 P52988 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&waitSearcher=true&wt=javabin} {commit=} 0 49
[junit4:junit4]   2> 261030 T790 C1447 P37291 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52742/ba/l/collection1/, StdNode: http://127.0.0.1:37291/ba/l/collection1/, StdNode: http://127.0.0.1:36784/ba/l/collection1/, StdNode: http://127.0.0.1:45463/ba/l/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 261033 T774 C1446 P52742 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 261034 T789 C1447 P37291 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 261034 T809 C1449 P36784 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 261035 T824 C1448 P45463 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 261053 T774 C1446 P52742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty1/index,segFN=segments_7,generation=7}
[junit4:junit4]   2> 261054 T774 C1446 P52742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4]   2> 261055 T774 C1446 P52742 oass.SolrIndexSearcher.<init> Opening Searcher@3a8e7e2b main
[junit4:junit4]   2> 261056 T774 C1446 P52742 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 261058 T782 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a8e7e2b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 261059 T774 C1446 P52742 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 27
[junit4:junit4]   2> 261073 T809 C1449 P36784 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty3/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 261074 T809 C1449 P36784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 261073 T824 C1448 P45463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371701366133/jetty4/index,segFN=segments_7,generation=7}
[junit4:junit4]   2> 261075 T824 C1448 P45463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4]   2> 261075 T809 C1449 P36784 oass.SolrIndexSearcher.<init> Opening Searcher@1731f6aa main
[junit4:junit4]   2> 261076 T809 C1449 P36784 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 261076 T824 C1448 P45463 oass.SolrIndexSearcher.<init> Opening Searcher@16403e6b main
[junit4:junit4]   2> 261077 T824 C1448 P45463 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 261078 T815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1731f6aa main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 261079 T836 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16403e6b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 261079 T809 C1449 P36784 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 45
[junit4:junit4]   2> 261080 T824 C1448 P45463 oasup.LogUpdateProcessor.finish [collection1] webapp=/ba/l path=/update params={commit=true&version=2&softCommit=false&expungeDeletes=false&waitSearcher=true&wt=javabin&commit_end_point=true} {commit=} 0 45
[junit4:junit4]   2> 261113 T789 C1447 P37291 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: com

[...truncated too long message...]

upressed...
[junit4:junit4]   1>     /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4]   1>     DATA: ...supressed...
[junit4:junit4]   1>     /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4]   1>     DATA: ...supressed...
[junit4:junit4]   1>     /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4]   1>     DATA: ...supressed...
[junit4:junit4]   1>     /solr/configs/conf1/currency.xml (0)
[junit4:junit4]   1>     DATA: ...supressed...
[junit4:junit4]   1>     /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4]   1>     DATA: ...supressed...
[junit4:junit4]   1>   /solr/overseer (3)
[junit4:junit4]   1>   DATA:
[junit4:junit4]   1>       
[junit4:junit4]   1>    /solr/overseer/queue (0)
[junit4:junit4]   1>    /solr/overseer/queue-work (0)
[junit4:junit4]   1>    /solr/overseer/collection-queue-work (0)
[junit4:junit4]   1>   /solr/collections (2)
[junit4:junit4]   1>    /solr/collections/collection1 (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/collection1/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89895820731023374-2-n_0000000004 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89895820731023372-5-n_0000000003 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89895820731023370-4-n_0000000002 (0)
[junit4:junit4]   1>     /solr/collections/collection1/shards (0)
[junit4:junit4]   1>     /solr/collections/collection1/leaders (1)
[junit4:junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "node_name":"127.0.0.1:36784_ba%2Fl",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:36784/ba/l",
[junit4:junit4]   1>            "core":"collection1"}
[junit4:junit4]   1>    /solr/collections/control_collection (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/control_collection/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/89895820731023363-1-n_0000000000 (0)
[junit4:junit4]   1>     /solr/collections/control_collection/shards (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "node_name":"127.0.0.1:52988_ba%2Fl",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:52988/ba/l",
[junit4:junit4]   1>            "core":"collection1"}
[junit4:junit4]   1>   /solr/aliases.json (0)
[junit4:junit4]   1>   /solr/clusterstate.json (0)
[junit4:junit4]   1>   DATA:
[junit4:junit4]   1>       {
[junit4:junit4]   1>         "collection1":{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{
[junit4:junit4]   1>                 "2":{
[junit4:junit4]   1>                   "collection":"collection1",
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:52742_ba%2Fl",
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:52742/ba/l",
[junit4:junit4]   1>                   "core":"collection1"},
[junit4:junit4]   1>                 "3":{
[junit4:junit4]   1>                   "collection":"collection1",
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:37291_ba%2Fl",
[junit4:junit4]   1>                   "state":"down",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:37291/ba/l",
[junit4:junit4]   1>                   "core":"collection1"},
[junit4:junit4]   1>                 "4":{
[junit4:junit4]   1>                   "collection":"collection1",
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:36784_ba%2Fl",
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:36784/ba/l",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "leader":"true"},
[junit4:junit4]   1>                 "5":{
[junit4:junit4]   1>                   "collection":"collection1",
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:45463_ba%2Fl",
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:45463/ba/l",
[junit4:junit4]   1>                   "core":"collection1"}}}},
[junit4:junit4]   1>           "router":"compositeId"},
[junit4:junit4]   1>         "control_collection":{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{"1":{
[junit4:junit4]   1>                   "collection":"control_collection",
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:52988_ba%2Fl",
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:52988/ba/l",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "leader":"true"}}}},
[junit4:junit4]   1>           "router":"compositeId"}}
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=BC868A7A59F8CA9D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en -Dtests.timezone=SystemV/YST9 -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE  102s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:52742/ba/l/collection1lastClient and got 252 from http://127.0.0.1:36784/ba/l/collection1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC868A7A59F8CA9D:3D6004622EA7AAA1]:0)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 340947 T738 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 101806 T737 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 342093 T830 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 342093 T830 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 342093 T830 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=115), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=115)), sim=DefaultSimilarity, locale=en, timezone=SystemV/YST9
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=2,free=170668544,total=322875392
[junit4:junit4]   2> NOTE: All tests run in this JVM: [PreAnalyzedUpdateProcessorTest, SolrIndexSplitterTest, OverseerTest, RecoveryZkTest, TestRandomDVFaceting, CSVRequestHandlerTest, LoggingHandlerTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 103.09s, 1 test, 1 failure <<< FAILURES!

[...truncated 751 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1252 tests, 1 failure, 16 ignored (7 assumptions)

Total time: 40 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure